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1  Executive  Summary 
1.1  Introduction 


Maintenance  Mentor  (MXM)  is  a  research  effort  conducted  by  a  joint  AFRL/HESR  and  Northrop 
Grumman  Information  Technology  team  (referred  to  throughout  this  report  simply  as  “the  team”  or  “the 
research  team”)  to  identify  the  basic,  high-level  requirements  necessary  for  a  follow-on  research  and 
development  program  aimed  at  improving  flightline  diagnostic  capabilities.  The  effort  was  initially 
focused  on  identifying  specific  informational  and  visual  requirements  and  needs  of  the  maintainor  for  a 
specific  avionics  diagnostics  task  on  a  specific  aircraft.  Soon  after  its  start,  however,  redirection  of  the 
project  began  to  move  it  from  a  single  task  approach  to  one  of  documenting  the  basic  flightline 
troubleshooting  and  maintenance  process  and  identifying  the  requirements  that,  if  satisfied,  could 
significantly  enhance  the  maintainers’  efficiency  and  effectiveness  in  turning  aircraft  mission-capable 
much  faster  than  is  currently  possible.  With  the  redirection  of  the  project,  the  initial  predisposition  to 
emphasize  a  single  weapon  system  approach  that  might  later  be  broadened  evolved  into  a  higher-level, 
multi-weapon  system  view.  This  program  was  felt  to  be  particularly  timely  since  statistics  clearly  show 
that  most  aircraft  fix  rates  have  not  markedly  improved  in  recent  years  and  many  have  declined.  Although 
this  is  certainly  a  multi-faceted  problem,  it  is  apparent  that  one  element  of  the  solution  is  that  the 
maintainer’s  ability  to  troubleshoot  and  diagnose  problems  requires  enhancement.  The  MXM  research  is 
aimed  at  helping  bring  that  about. 

1.2  System  Identification 

One  of  the  very  early  phases  of  the  MXM  effort  was  System  Identification.  Simply  stated.  System 
Identification  was  a  basic  process  through  which  the  team  sought  to  identify  a  target  weapon  system 
(aircraft)  and  a  target  subsystem  upon  which  to  focus  their  research.  Because  of  re-direction  of  the 
project,  it  was  decided  that  selections  for  target  weapons  systems  and  subsystems  would  be  conducted  in 
parallel  to  provide  maximum  flexibility  despite  the  direction  the  program  later  took. 


It  was  decided  that  the  A-10,  B-l,  B-52,  F-15,  F-16,  KC-135,  C-130,  C-5  and  C-17  weapon  systems 
would  be  considered.  For  a  number  of  reasons  detailed  in  the  report,  low  density,  high  demand  systems 
(i.e.,  F-l  17,  B-2,  Rivet  Joint,  AWACS,  Joint  Stars,  Special  Operations)  and  pure  training  systems  such  as 
the  T-l,  T-37,  and  T-38  were  not  considered.  Likewise,  developmental  systems  such  as  the  F/A-22,  F-35, 
and  CV-22  were  not  considered  in  the  assessment. 
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AFRL/HESR  set  out  to  gather  the  maintenance  data  that  would  be  analyzed  to  support  System 
Identification.  They  successfully  collected  most  of  the  Air  Combat  Command  (ACC)  data  that  was 
needed  but  getting  Air  Mobility  Command  (AMC)  proved  more  difficult  and  resulted  in  only  some  basic 
information  becoming  available.  In  the  interest  of  time,  the  team  decided  to  move  forward  without  going 
after  additional  information. 

The  team  devised  some  basic  guidelines  to  provide  structure  for  the  decision  process.  Those  guidelines 
are  listed  below: 

♦  High  Maintenance  Man-hour  Consumer 

♦  Probability  of  Data  Availability 

♦  Probability  of  Access  for  Data  Gathering/Knowledge  Acquisition 

♦  Potential  for  SPO  Support 

♦  Ease  of  Interfacing  with  the  Aircraft 

♦  Potential  for  Major  Command  (MAJCOM)  Support 

♦  Longevity/Potential  Payback 

♦  Other  Programs  On-going 

Scores  ranging  from  1  to  5  were  assessed  by  the  team  to  denote  die  relative  ease  (1)  or  the  (5)  difficulty  in 
dealing  with  the  area  covered  by  the  appropriate  guideline.  Under  “High  Maintenance  Man-hour 
Consumer,”  for  example,  if  an  aircraft  was  considered  to  be  a  very  high  consumer  of  maintenance  time 
according  to  the  various  statistics  and  inputs  the  team  had  available  to  them  a  score  of  5  was  assessed.  A 
very  low  consumer  received  a  1,  while  scores  between  the  two  extremes  were  assessed  based  on  the 
information  available.  Although  a  numeric  score  was  not  given  for  “Other  Programs  On-going,”  if  the 
team  was  aware  of  other  programs  that  might  conflict  with  the  MXM  effort  or  compete  with  it  or  a 
potential  6.3  follow-on  program  for  support  that  fact  was  noted. 

The  F-15,  F-16,  KC-135,  C-130,  and  C-5  were  the  top  five  weapons  systems  based  on  the  scores  attained. 
The  team  decided  to  focus  on  the  F-15  and/or  F-16  and  the  C-130  and/or  KC-135. 

Subsystem  selection  was  done  in  much  the  same  way  as  the  weapon  system  selection.  ACC  data  on  the 
A-10,  B-l,  B-52,  F-15,  F-16,  and  EC/HC-130  data  was  used  and  assessments  were  made  of  the  primary 
subsystems  listed  below  in  each  of  those  aircraft. 

♦  Fuel 

♦  Hydraulic 
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♦  Propulsion 

♦  Landing  Gear 

♦  Flight  Controls 

♦  Radar 

♦  Electronic  Countermeasures 

♦  Electrical 

Each  of  the  subsystems  on  the  aircraft  in  question  was  evaluated  using  the  guidelines  listed  below. 

♦  Top  Five  Man-hour  Consumer 

♦  Ease  of  Troubleshooting 

♦  Relevance  to  the  MXM  Project 

♦  Potential  Integration  Challenges 

♦  Potential  Improvement  for  Maintained 

As  in  weapon  system  selection  scores  ranging  from  1  through  5  were  assessed  with  the  exception  of  “Top 
Five  Man-hour  Consumer.”  In  this  case,  if  the  subsystem  had  been  one  of  the  top  five  maintenance  man¬ 
hour  consumers  over  the  past  year  a  score  of  5  was  awarded.  If  it  had  not  been,  a  score  of  1  was  given. 
This  is  the  only  category  in  which  no  intermediate  scores  were  used  and  it  was  also  the  only  category  in 
which  the  score  was  purely  objective. 

Table  2  in  the  report  shows  that  flight  controls,  propulsion,  fuel,  landing  gear,  and  electrical  garnered  the 
highest  scores.  Flight  controls  was  the  subsystem  on  which  the  team  ultimately  decided  to  center  the 
MXM  research.  That  decision  was  made  not  only  because  flight  controls  attained  the  highest  overall  score 
across  all  weapons  systems  and  was  also  a  high  scorer  in  the  F-15,  F-16  and  C-130  target  weapon 
systems,  but  for  a  number  of  other  reasons  as  well,  all  of  which  are  spelled  out  in  the  report. 

1.3  Requirements  Definition 

This  project  phase,  consisting  of  three  interrelated  tasks,  addressed  maintained'  flightline  work  in  light  of 
required  data  and  information.  These  three  tasks  -  Knowledge  Acquisition  (KA),  Cognitive  Task  Analysis 
(CTA),  and  Information  Resources  Analysis  (IRA)  -  were  conducted  with  a  scope  of  reference  modified 
from  that  originally  planned.  Starting  at  the  kickoff,  AFRL/HESR  raised  the  issue  of  expanding  the  scope 
of  research  to  address  multiple  aircraft  so  as  to  obtain  results  more  generally  applicable  and  informative 
than  details  on  one  aircraft  and  one  weapon  system.  After  discussions  among  all  parties,  the  MXM  team 
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agreed  on  a  new  scope  based  on  three  criteria:  First,  the  scope  of  system/subsystem  concern  would  be  a 
general  class  of  apparatus  applicable  to  a  wide  variety  of  operational  USAF  aircraft.  Second,  this  scope  of 
concern  was  agreed  to  be  flight  controls.  A  reasonable  range  of  operational  USAF  aircraft  would  be 
reviewed  with  respect  to  flight  control  maintenance  issues.  Third,  the  MXM  team  agreed  to  achieve  this 
breadth  of  range  by  reviewing  both  small  aircraft  (i.e.,  fighters)  and  large  aircraft  (airlift  aircraft  in  this 
case). 

Knowledge  Acquisition  (KA)  is  the  process  of  obtaining  task-related  knowledge  from  subject  matter 
experts  (SMEs)  and  other  information  resources.  In  this  case,  the  team  sought  data  on  the  cognitive  and 
informational  aspects  of  maintenance  functions  for  the  population  of  aircraft  surveyed.  The  KA  plan  was 
tailored  to  emphasize  topics  common  across  the  set  of  aircraft  and  maintained.  To  achieve  this  breadth 
the  team  conducted  detailed  data  gathering  and  interviews  with  a  set  of  maintainer  SMEs  reflecting  a 
reasonable  sample  of  different  aircraft  and  aircraft  types. 

Planning  the  KA  effort  required  considerable  thought  and  discussion.  Site  access  was  a  problem  due  to 
Operation  Enduring  Freedom  (OEF),  Operation  Iraqi  Freedom  (OIF),  and  ongoing  Homeland  Defense 
commitments.  It  was  concluded  toe  most  constructive  and  feasible  course  of  action  would  be  a  two-phase 
KA  itinerary.  The  first  phase  would  address  fighter  aircraft.  Nellis  AFB,  Nevada  was  identified  as  a 
lucrative  KA  site  for  a  variety  of  fighter  aircraft  including  A-lOs,  F-15Cs,  F-15Es,  and  F-16C/Ds,  as  well 
as  toe  F/A-22  and  RQ-1.  The  second  phase  would  address  an  airlift  aircraft.  It  was  decided  that  a  visit  to 
Charleston  AFB,  SC  for  KA  on  toe  C-17  was  toe  best  feasible  option.  A  four-person  KA  team  executed 
this  itinerary  during  April  2003. 

The  multi-aircraft  scope  required  adjustments  to  toe  KA  tactics.  The  most  commonly  used  on-site  KA 
tactics  include  structured  interviews  with  SMEs,  observations  of  target  tasks  being  accomplished,  and 
task  simulations.  The  shift  to  a  more  general  scope  induced  several  new  or  newly-prioritized  criteria  for 
choosing  toe  specific  KA  tactics.  Emphasis  was  placed  on  maximizing  on-site  time  while  minimizing 
digressions  into  data  too  fine-grained  for  toe  scope  of  toe  effort.  In  addition,  toe  team  had  to  maximize  the 
“generalizability”  of  toe  data  acquired  on  task  processes,  information  requirements,  and  toe  tools  and 
support  aids  employed  in  toe  maintenance  work. 

Proactive  SME  review  of 'straw  man'  materials  (as  contrasted  with  passive  free-form  questioning)  was 
given  priority  in  order  to  accelerate  model  formation  for  process  and  information  needs.  The  target  SME 
population  was  expanded  to  include  technical  support  and  other  relevant  personnel  to  get  a  better 
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overview  of  the  maintenance  environment.  A  KA  plan  was  then  developed  to  serve  as  a  structured  guide 
for  the  team. 

The  KA  plan  used  a  four  phase  “cascade”  approach  to  exploring  maintenance  process  and  associated  task 
elements.  First,  a  general  model  of  the  maintenance  process  flow  (from  problem  notification  through  to 
returning  the  aircraft  to  duty)  was  elicited,  refined,  and  validated  with  subsequent  SMEs.  Second,  this 
model  was  leveraged  as  the  framework  for  eliciting  and  collating  comments  on  procedures,  practices,  and 
problems  related  to  each.  Third,  this  model  (and  the  data  on  procedural  matters)  was  then  used  as  the 
framework  for  polling  SMEs  on  what  data  or  information  they  needed  or  expected  to  employ  at  each 
stage  of  the  process  path  and  with  regard  to  the  procedural  issues.  In  the  fourth  phase  of  this  exploration, 
the  team  visited  selected  support  sections  to  review  flight  control  tools  and  instruments  and  to  gather  Hnta 
on  the  tools  and  their  usage. 

A  meeting  room  was  made  available  to  the  KA  team  at  both  sites,  and  all  SME  interviews  were  conducted 
there  in  accordance  with  an  agenda  prepared  by  host  unit  representatives.  Each  scheduled  session 
typically  afforded  at  least  90  minutes  with  each  SME  group.  Sessions  began  with  the  team  introducing 
themselves,  the  MXM  project,  and  the  session  objectives.  A  lead  interviewer  facilitated  each  session,  with 
the  other  team  members  participating  as  circumstances  warranted.  The  set  of  key  issues  assembled  prior 
to  the  KA  visits  was  used  as  the  basis  for  toe  interviews.  Significant  points  arising  in  earlier  interviews 
were  fed  forward  as  talking  points  in  subsequent  sessions. 

The  team  began  toe  first  phase  exploration  of  the  maintenance  process  path  by  presenting  a  straw  man 
sequence  of  steps  for  a  representative  flight  control  maintenance  process.  The  SMEs  were  then  asked  for 
their  opinions  and  invited  to  make  any  comments  or  corrections  they  thought  would  make  it  more 
accurate.  Once  this  review  and  modification  cycle  was  completed,  toe  revised  model  (depicted  in  Figure  1 
in  toe  report)  was  reviewed  and  validated  by  subsequent  SME  groups.  The  model  was  effectively 
validated  with  toe  first  session  on  toe  first  day,  which  allowed  it  to  be  leveraged  for  toe  other  KA  goals  at 
a  very  early  point.  The  final  process  path  model  breaks  out  into  eight  primary  steps  or  phases  as  follows: 

♦  Problem  identification/reporting 

♦  Front-end  unit  coordination  (to  get  toe  aircraft  into  toe  maintenance  process) 

♦  Maintenance  setup/preparation 

♦  Troubleshooting  cycle  (subsuming  testing,  diagnosis,  prognosis,  repair  actions,  and  testing/evaluation 
to  toe  point  toe  work  is  deemed  complete) 

♦  Solution  reporting/documentation 
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♦  Solution  validation/verification  and  process  completion  decision 

♦  Maintenance  stand-down/cleanup 

♦  Back-end  unit  coordination  (to  get  the  aircraft  back  to  duty) 

The  process  path  represents  the  entire  end-to-end  progression  for  a  typical  maintenance  cycle,  the 
completion  and  documentation  of  which  is  captured  in  aggregate  performance  statistics.  The  KA  t^am 
discovered  a  substantial  proportion  of  apparent  flight  control  discrepancies  that  were  resolved  without 
going  through  this  process  path  or  being  documented.  This  means  that  maintainer  time  and  effort  is 
expended  on  diagnosis  and  problem  solving  that  is  not  reflected  in  the  maintenance  statistics.  This  issue 
arose  in  the  very  first  KA  session,  and  it  was  added  as  a  talking  point  for  subsequent  interviews.  Up  to 
50%  of  die  flight  control  discrepancies  in  some  aircraft  were  said  to  fall  into  this  “no  fix"  category  owing 
to  several  factors.  For  example,  this  situation  was  sometimes  attributed  to  “switchology”  (incorrect 
control,  switch,  or  knob  settings)  or  to  resetting  some  component  of  die  flight  control  (subsystem  or  an 
associated  (sub)system.  In  other  cases,  reported  flight  control  discrepancies  could  not  be  duplicated  or 
could  be  readily  explained  by  incorrect  pilot  actions. 

Rapidly  arriving  at  a  consensus  process  path  model  allowed  the  team  to  correlate  information 
requirements  with  process  path  steps  early  in  the  KA  effort.  The  approach  was  to  step  the  SMEs  through 
the  acknowledged  process  path  map  and  probe  at  each  step  for  the  key  issues  they  needed  to  resolve  to 
accomplish  that  step.  Section  4.1.6  of  the  report  includes  a  summary  listing  of  the  most  commonly  cited 
information  requirements  for  each  step  in  the  flight  control  maintenance  process  path. 

Several  issues  and  concerns  regarding  current  flight  control  maintenance  information  resources  arose 
during  the  interview  sessions.  The  maintainer  SMEs  consistentiy  made  reference  to  problems  with 
operational  reference  aids  (e.g.,  manuals,  checklists,  TOs).  For  example,  the  SMEs  cited  differences 
between  pilot  checklists  and  maintenance  TOs  as  sources  of  some  of  the  "no  fix"  situations  mentioned 
earlier.  The  team  paid  particular  attention  to  the  availability,  utility,  and  sufficiency  of  diagnostic  decision 
aids.  Although  much  time  was  spent  probing  for  data  on  deficiencies  in  the  diagnostic  aids,  the  SMEs 
offered  relatively  few  complaints  on  the  aids  themselves.  In  general,  they  emphasized  the  advantages  of 
having  aids  that  are  employed  along  with  built-in  test  (BIT)  code  data,  which  greatly  expedites  diagnosis. 
Both  paper  fault  isolation  guides  (FIs)  and  available  electronic  diagnostic  aids  have  been  found  to  contain 
deficiencies,  gaps,  and  even  some  errors.  The  lack  of  feedback  and  the  slow  revision  process  for  TOs  and 
FIs  were  recurrently  cited  as  problems. 
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Interestingly,  those  maintainer  groups  with  equivalent  access  to  both  paper  and  electronic  fault  diagnosis 
aids  indicated  a  general  preference  for  the  paper  materials.  This  preference  derived  in  part  from  the 
problems  unique  to  the  electronic  aids.  Because  electronic  aids  require  laptops  (or  similar  devices)  on 
which  to  run  applications,  access  to  the  aid  depends  on  availability  and  serviceability  of  these  platforms. 
Maintenance  work  is  often  two-handed  work.  This  means  that  maintainers  must  set  a  laptop  somewhere 
and  then  go  back  and  forth  between  manual  actions  on  the  aircraft  and  manual  interactions  with  the 
laptop.  At  the  very  least,  much  head  turning  is  required  to  address  both  the  aircraft  and  the  laptop.  This 
divides  attention  and  consumes  time. 

The  legibility  of  laptop-based  aids  varies  with  circumstances  and  often  diminishes  usability.  For  example, 
severe  glare  off  the  display  screen  under  certain  lighting  conditions,  especially  direct  sunlight,  makes  it 
unreadable.  Another  problem  is  the  usage  of  small  fonts  making  it  difficult  to  read  text  information  or 
diagram  captions.  Reading  such  small  fonts  requires  the  user  to  get  close  to  the  display  (further  increasing 
the  probability  he/she  cannot  use  the  aid  while  working  directly  on  the  aircraft).  The  procedural  and 
interface  structure  of  the  laptop-based  diagnostic  aids  is  such  that  it  can  be  slower  to  navigate  through  the 
fault  tree  on  the  laptop  than  using  a  reference  card.  In  other  words,  the  laptop-based  aids  may  prove  to  be 
cumbersome  logically  as  well  as  physically.  Version  discrepancies  among  software-based  reference  and 
diagnostic  aids  are  also  a  persistent  irritant. 

These  types  of  problems  explain  the  SMEs’  lack  of  optimism  for  electronic  documentation  aiding.  Most 
documentation  is  still  done  by  hand  with  paper,  such  as  AFTO  Form  781A  entries,  logbook  entries,  and 
notes  on  clipboards.  This  documentation  is  generated  in  the  course  of  die  maintenance  process  in  the 
actual  maintenance  setting.  As  a  result,  using  a  computer  for  these  documentation  functions  is  subject  to 
the  same  usability  problems  noted  for  computer-based  diagnostic  aids.  When  asked  if  they  saw  any 
potential  for  improved  support  by  computerizing  incidental  documentation,  the  SMEs  consistently 
answered  in  the  negative. 

The  SMEs  were  also  asked  about  their  tools  and  instruments.  Surprising,  the  frequency  and  criticality  of 
such  problems  surpassed  those  reported  for  the  diagnostic  guides.  Maintainers  do  not  have  much 
confidence  in  some  of  their  test  equipment.  Much  of  this  equipment  is  old  and/or  dysfunctional,  and  some 
of  the  newer  equipment  is  less  usable  or  robust  than  the  older  alternatives. 

A  surprising  equipment  issue  was  the  extent  to  which  connectors  and  connections  (as  contrasted  with  the 
components  being  connected)  cause  headaches  for  maintainers.  Many  problems  center  on  connections 
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between  aerospace  ground  equipment  (AGE),  such  as  hydraulic  mules,  and  the  aircraft.  In  addition, 
problems  with  cables  and  connectors  can  affect  test  readings  and  mislead  troubleshooters  (at  the  cost  of 
time  and  effort).  For  example,  the  TT-205  tester  connection  set  (“hose  kit”)  is  the  most  common  obstacle 
to  using  the  tester,  and  the  lack  of  compatible  connectors  is  a  major  reason  why  the  TT-205  is  not  used  on 
the  F-16. 

The  notion  of*  expertise  ’  is  important  in  undertaking  task-specific  cognitive  and  information  requirement 
analyses.  It  is  important  to  identify  what  features  are  associated  with  attributed  expertise  in  a  given  task 
The  SMEs  provided  several  features  characteristic  of  expert  maintainer  performance. 

It  was  learned  that  a  variety  of  distinctions  between  experts  and  novices  not  only  affect  the  quality  and 
course  of  the  maintenance  process,  but  also  affect  the  readiness  with  which  other  parties  accept  their 
diagnoses.  Novices  tend  to  scrupulously  follow  the  procedures  step-by-step  and  rely  exclusively  on  the 
technical  order  (TO)  and  BIT  as  the  entirety  of  their  diagnostic  approach  (as  opposed  to  digging  into  the 
fault  via  exploratory  troubleshooting).  Experts  are  more  proficient  about  relating  symptoms  to  states  or 
features  of  the  physical  aircraft  than  novices — a  capability  repeatedly  described  as  reflecting  experts’ 
better  knowledge  of  how  the  aircraft  and  its  constituent  subsystems  actually  work.  Maintained,  technical 
support  staff,  and  trainers  agreed  that  novices  are  more  reluctant  to  initiate  free  form  exploratory 
troubleshooting  when  the  available  diagnostic  procedures  fail  to  pinpoint  the  fault.  This  makes  novices 
quicker  and  more  amenable  to  stopping  the  maintenance  process  and  “making  the  call”  (invoking  external 
technical  support)  once  the  cookbook  procedure  bogs  down. 

This  naturally  leads  to  the  subject  of  training.  The  team  made  a  point  to  interview  trainers,  and  training 
was  cited  repeatedly  in  the  other  SME  sessions.  Much  of  the  discussion  on  training  improvements 
mirrored  the  types  of  points  made  with  respect  to  perceived  differences  between  novice  and  expert 
maintainer  capabilities. 

In  each  interview  session,  SMEs  were  asked  what  type(s)  of  data  or  information  would  most  improve 
their  ability  to  perform  flight  control  maintenance.  Throughout  all  the  interview  sessions,  the  #1  item  on 
the  SME  “wish  list”  was  better  information  on  what  was  happening  in  flight  when  a  flight  control  issue 
occurred. 

The  #2  recommendation  from  the  SMEs  was  better  capability  to  simulate  in-flight  conditions  on  the 
ground.  Yet  another  information  need  concerns  knowledge  that  is  available,  but  not  effectively  captured 
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and  disseminated.  In  spite  of  the  generally  voluminous  “official”  data  found  in  the  TOs,  there  is  much 
relevant  and  useful  information  that  can  only  be  obtained  from  maintenance  experience  with  a  particular 
aircraft.  Such  information  includes  tips,  tricks  of  the  trade,  and  illustrative  “lore”  derived  from 
experience.  The  SMEs  uniformly  cited  the  value  of  the  unit  logbooks  as  information  resources;  however, 
there  are  no  effective  channels  for  general  discussion  and  note-sharing  (e.g.,  chat  rooms;  bulletin  boards, 
ListServ  forums). 

The  KA  visits  to  Nellis  and  Charleston  went  very  well,  and  the  team  obtained  considerable  data  in  a 
relatively  short  length  of  time.  The  attention  to  generating  a  KA  plan  in  advance  of  the  trips  allowed  the 
team  to  coordinate  its  effort  and  to  make  maximum  use  of  on-site  time.  The  “cascade  approach”  to 
incrementally  fleshing  out  team  understanding  of  flight  control  maintenance  atop  the  initial  process  path 
map  allowed  the  team  to  remain  focused  throughout  the  KA  effort  Attention  to  gathering  data  on 
auxiliary  units  of  the  maintenance  organization  (i.e.,  test  stations  and  support  sections)  enabled  the  team 
to  understand  the  core  maintenance  process  in  a  wider  context.  The  data  presented  above  represents  only 
a  portion  of  what  was  obtained.  However,  even  if  this  were  all  that  was  gathered  on  the  two  KA  trips,  the 
outcome  would  still  have  to  be  considered  a  solid  success. 

Cognitive  Task  Analysis  (CTAl  concerns  the  examination  and  critical  analysis  of  a  work  activity  or 
process  with  regard  to  the  cognitive  aspects  of  work.  Basically,  this  means  analyzing  the  “mental  work” 
associated  with  a  given  task  in  the  same  way  that  older  methodologies  analyzed  that  task’s  “physical 
work.”  A  task’s  “cognitive  aspects”  commonly  include: 

♦  The  perceptual  acquisition  of  data  in  the  course  of  a  task 

♦  The  data  and  information  elements  critical  to  conducting  the  given  task 

♦  Mental  models  a  worker  employs  for  the  task  process  itself  and  the  subject  matter  he/she  must 
address  during  the  task 

♦  The  decisions  that  must  be  made  to  complete  the  task 

♦  The  critical  dimensions  of  decisions  made  in  the  task  (e.g.,  critical  data,  time  to  decide,  confounding 
factors,  mode  of  inference,  means  for  testing  alternatives) 

♦  The  degree  of  “cognitive  workload”  or  “cognitive  burden”  entailed  in  performing  the  task 

♦  Cognitive  and  informational  factors  which  can  induce  errors  and  other  degradations  in  task 
performance  (e.g.,  data  deficiencies,  data  overload) 

The  MXM  effort  was  geared  to  explore  the  relationships  between  maintainer  information  processes  and 
their  maintenance  tasks.  The  majority  of  the  points  cited  in  the  section  on  KA  relate  to  data,  information 
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resources,  and  process.  Applying  all  this  information  to  the  maintenance  task  requires  stating  what  it  is 
about  task  performance  one  is  seeking  to  analyze.  In  the  course  of  the  KA  work  the  team  obtained 
summary  statistics  on  USAF  maintenance  performance.  A  set  of  tabular  compilations  of  such  data  (for 
fighter  aircraft)  is  offered  in  Appendix  I.  A  more  concise  summary  table  of  performance  statistics  for  four 
categories  of  fighter  aircraft  covered  in  the  KA  work  is  given  in  Table  6. 

There  are  a  number  of  points  in  this  data.  In  absolute  terms,  4-  and  8-hour  fix  rate  performance  has 
degraded  for  all  aircraft  over  the  10-year  period.  With  the  exception  of  the  A-10,  performance  results  fall 
short  of  established  standards  for  the  10-year  period  as  a  whole.  There  was  no  clear  pattern  to  the 
summary  outcomes  with  respect  to  average  performance  (relative  to  standard)  over  the  10-year  period. 

Because  these  particular  statistics  are  framed  with  regard  to  time,  it  is  reasonable  to  characterize  the 
evaluation  context  as  temporal.  The  fact  that  the  4-hour  rate  performance  has  degraded  more  than  the 
8-hour  rate  performance  is  consistent  with  a  situation  in  which  the  maintenance  process  is  taking  longer 
and  longer  to  effectively  complete.  Time  to  completion  is  the  one  general  criterion  that  can  be  interpreted 
to  subsume  negative  effects  resulting  from  a  variety  of  possible  sources,  such  as  errors  and  grappling  with 
ambiguity.  Given  its  generality  and  its  prominence  in  the  available  data,  it  is  therefore  reasonable  to  adopt 
temporal  maintenance  process  performance  as  the  general  dimension  for  framing  the  analysis  of  the 
maintenance  process. 

The  crux  of  the  CTA  effort  was  to  identify  a  model  or  schema  for  coherently  laying  out  the  flight  control 
maintenance  process  path.  The  model  or  framework  judged  most  amenable  to  MXM  purposes  was  the 
OODA  Loop  of  Col.  John  R.  Boyd  (Boyd,  1987).  The  OODA  Loop  is  a  cyclical  interrelating  process  path 
leading  from  perception  of  situational  data  through  decision  making  and  on  to  resultant  action.  An  OODA 
Loop  affords  a  scope  of  referential  context  identical  to  that  of  MXM  (maintained  perceiving  data  and 
making  decisions  for  ongoing  maintenance  actions).  This  permits  one  to  proceed  without  decomposing 
the  subject  process  into  multiple  and  potentially  irreconcilable  sub-models. 

The  mapping  of  the  maintenance  process  path  onto  an  OODA  representation  is  presented  in  the  form  of 
1 1  tables  in  Section  4.2.5.  Each  of  the  tables  provides  a  cursory  enumeration  of  elements  associated  with 
each  of  the  four  OODA  steps.  For  the  “Observe”  step,  these  will  be  elements  (e.g.,  data,  situations)  that 
must  be  perceived.  For  the  “Orient”  step,  these  will  be  elements  for  which  maintained  must  achieve 
situational  awareness.  For  the  “Decide”  step,  these  will  be  the  issues  or  topics  for  which  decisions  must 
be  made.  Finally,  for  the  “Act”  step,  these  will  be  the  typical  actions  (or  courses  of  action)  deriving  from 
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that  phase’s  “Decide”  step.  As  applicable,  the  tables  also  include  a  listing  of  the  potential  “leaps”  (process 
path  jumps  other  than  simply  moving  forward  to  the  next  step)  that  might  occur. 

In  addition,  an  inventory  of  the  most  significant  cognitive  performance  issues  identified  from  the  SME 
sessions  was  compiled,  and  several  pages  in  Section  4.2.6  are  dedicated  to  a  detailed  discussion  of  them. 
Those  issues  include:  complexity  in  the  flight  control  maintenance  decision  space;  criticality  of  up-front 
information;  criticality  of  simulating  or  reproducing  the  perceived  problem;  fault  sources  unaccounted  for 
in  available  diagnostic  aid  representations;  progressive  deskilling  in  the  maintainer  population;  and  value 
of  experiential  knowledge. 

Information  Requirements  Analysis  (IRA)  is  the  examination  and  analysis  of  a  work  activity  or  process 
with  regard  to  the  information  set  required  to  perform  a  task,  the  set  of  data  and  information  available 
during  the  task,  the  differences  between  these  two  sets,  and  the  implications  of  these  differences  on  task 
support. 

One  of  die  most  salient  characteristics  of  the  maintenance  process  is  that  it  is  collaborative;  that  is,  it 
involves  multiple  players  jointly  working  to  diagnose  and  repair  the  reported  fault.  It  is  not  the  case  that 
all  possible  players  participate  in  all  reported  cases.  Some  sources  of  data  and  expert  knowledge 
participate  only  if  invoked  by  the  front  line  maintained.  This  constitutes  a  discretionary  'reachback'  for 
relevant  information  as  circumstances  warrant.  Although  the  precise  details  vary  from  aircraft  to  aircraft, 
the  SMEs  outlined  three  representative  steps  in  a  progression  of  information  reachback.  The  first  is  to  on- 
base  technical  support  personnel,  such  as  Air  Force  Engineering  and  Technical  Service  (AFETS) 
personnel  and  forward-deployed  contractors.  The  second  is  to  remote  (off-base)  technical  expertise 
accessible  via  telecommunications  (e.g.,  hot  line).  The  third  is  to  technical  expertise  accessible  by  calling 
on  personnel  at  the  relevant  depot.  Each  of  these  reachback  assets  possesses  technical  data  resources 
exceeding  the  scope  and/or  depth  of  those  available  on  the  flightline.  The  general  form  of  this  reachback 
progression  is  illustrated  in  Figure  3. 

With  regard  to  maintenance  performance,  the  key  point  is  that  each  information  reachback  action  serves 
as  a  drag  on  the  maintainer’s  ability  to  resolve  the  reported  maintenance  problem  within  a  given 
timeframe.  Reachback  degrades  temporal  performance  metrics  for  a  number  of  reasons.  For  one  thing, 
setting  aside  the  maintenance  activity  to  call  in  outside  help  may  involve  practical  actions  (e.g.,  parking, 
putting  away  tools,  etc.)  that  either  consume  time  or  mandate  time  consumption  for  getting  back  on  task. 
Time  is  consumed  in  making  effective  contact  with  the  next  reachback  resource  and  in  relating  the 
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maintenance  problem  and  the  action  taken  so  far  to  the  reachback  resource.  Where  personal  contact  on¬ 
site  is  needed,  there  will  be  time  consumed  in  coordinating  such  consultations  and  there  may  be  redundant 
time  consumption  for  briefing  the  technical  support  staff  once  they  arrive. 

The  primary  information  needs  identified  for  flight  control  maintenance  are: 

♦  Detailed  and  comprehensive  description  of  die  perceived  problem  as  it  was  encountered  during  flight 
(the  SMEs’  #1  wish  list  item) 

♦  Access  to  experiential  knowledge  to  fill  in  the  gaps  in  the  formal  information  base 

♦  Capture  and  dissemination  of  experiential  knowledge  to  help  bring  less  experienced  maintainers  up  to 
more  expert  levels  of  performance 

♦  Dissemination  of  experiential  knowledge  across  maintenance  units  to  minimize  die  need  to  “reinvent 
the  wheel”  in  developing  tips  and  tricks  of  die  trade 

♦  Attention  to  real-world  operational  context  in  training  materials  and  courses 

♦  Better  background  technical  knowledge  for  incoming  trainees 

♦  Better  knowledge  of  an  aircraft’s  inner  workings  to  provide  less  experienced  maintainers  with  a 
foundation  for  analyzing  the  data  their  reference  and  diagnostic  aids  provide  them 

The  SME  groups  clearly  and  uniformly  cited  better  situation  awareness  on  in-flight  fault  context  as  their 
#1  desired  improvement.  Modem  military  aircraft  are  complex  systems.  As  a  result,  the  decision  space  for 
diagnosis  is  complex.  Any  and  all  clues  available  up  front  allow  the  maintaineifs)  to  more  expeditiously 
vector  in  on  a  likely  diagnosis  and  proceed  toward  resolution  or  repair.  The  most  straightforward 
approach  to  maximizing  such  up-front  SA  is  to  capture  and  record  as  much  in-flight  data  as  possible.  The 
maintainers  who  seemed  most  content  with  their  current  or  prospective  in-flight  data  access  were  those 
working  with  the  C-17,  RQ-1,  and  the  F/A-22 — all  of  whom  enjoy  significant  access  to  in-flight  data. 

Multiple  SMEs  involved  with  the  older  fighters  touted  the  utility  of  data  “snapshots”  (pilot-triggered 
recordings  of  selected  system  data).  For  example,  the  A- 10  SMEs  claimed  their  snapshot  access  was  a  big 
help  in  analyzing  reported  problems,  and  the  F-15E  maintainers  were  happy  that  a  snapshot  capability 
was  finally  being  made  available  on  their  aircraft.  Many  of  the  snapshot  capabilities  have  to  be  actively 
triggered  by  the  pilot,  who  may  be  preoccupied  at  the  point  a  flight  control  anomaly  occurs.  The  ultimate 
solution  is,  of  course,  to  dynamically  record  as  full  a  record  of  in-flight  data  as  the  available  hardware 
permits.  Though  not  cheap  in  absolute  terms,  it  would  seem  proportionally  cost-effective  to  consider 
adding  a  more  comprehensive  onboard  data  capture  and  recording  capability  to  these  older  aircraft. 
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One  alternative  aiding  strategy  might  be  to  evaluate  the  relative  incidence  of  candidate  fault  conditions 
and  provide  a  statistical  basis  for  “playing  the  odds”  when  initially  hypothesizing  what  a  given  fault  may 
be.  However,  good  information  up  front  is  more  likely  to  improve  overall  process  path  performance  than 
advice  on  the  odds  of  it  turning  out  to  be  this  or  that  based  on  prior  cases.  In  any  case  the  prospects  for 
compiling  sufficiently  accurate  statistical  data  are  diminished  so  long  as  the  present  proportion  of 
discrepancies  continue  to  go  unreported. 

Everyone  felt  a  need  for  better  capture  and  dissemination  of  experiential  knowledge,  but  just  as  uniformly 
noted  that  there  are  few  if  any  mechanisms  in  place  for  capturing  such  knowledge  and  even  fewer 
channels  for  disseminating  it.  There  are  two  straightforward  innovation  paths  that  would  reasonably 
address  this  deficiency: 

♦  Improve  capture  and  collation  of  logbook  entries 

♦  Provide  effective  channels  for  disseminating  access  to  capture  experiential  knowledge 

The  prospects  for  addressing  novice  deficiencies  in  background  knowledge  fall  outside  this  project’s 
purview.  Because  the  general  and  aircraft-specific  technical  knowledge  deficiencies  are  clearly  subjects  to 
be  addressed  with  training,  they  lie  outside  the  scope  of  the  central  focus  of  this  research — the 
maintenance  process  itself. 

1.4  Technology  Search 

A  general  search  was  performed  to  determine  what  tools  were  available  or  in  production  that  might 
potentially  suit  the  stated  needs  of  flightline  maintainers.  The  initial  1360  “hits,”  most  of  which  were 
eliminated  almost  immediately,  were  ultimately  reduced  to  29  that  were  included  even  if  there  was  a 
remote  possibility  of  a  match.  A  technical  review  identified  five  that  appeared  to  meet  some  requirements; 
however,  it  should  be  noted  that  the  information  obtained  was  gathered  from  marketing  material.  The 
requirements  against  which  the  technologies  in  this  research  were  validated  are  high  level  and  thus 
extremely  general.  On  the  other  hand,  Commercial-Off-The-Shelf  (COTS)  products  are  generally 
designed  to  meet  a  specific  need  within  a  specific  set  of  business  processes  and  a  standard  operational 
environment.  The  focus  of  these  products  is  typically  quite  narrow  and  well  defined. 

MXM  covers  a  very  broad  scope  of  requirements  and  the  solution  must  function  within  a  set  of  unique 
processes  and  run  on  a  variety  of  machines  with  interaction  to  many  outside  sources.  It  must  do  all  this 
reliably  in  a  wide  range  of  operational  environments  where  even  minor  “glitches”  may  often  be  out  of  the 
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question.  It  is  unlikely  that  any  COTS  product  will  meet  the  MXM  requirement.  Thus,  the  research  team 
recommended  that  a  custom  development  effort  be  initiated. 

To  provide  high  level  system  requirements  needed  to  facilitate  a  broad  search  for  potential  tools  the 
MXM  Team  hosted  a  Decision  Criteria  Development  Workshop  30-3 1  July  2003. 

In  that  two-day  period  the  team  facilitated  the  attendee  group  through  a  structured  process  that  resulted  in 
the  development  of  five  decision  models  (see  Table  25)  that  identify  the  basic  high  level  requirements  for 
a  potential  MXM  solution  system.  A  set  of  associated  requirements  and  criteria  were  developed  for  each 
model.  Table  3 1  depicts  the  combined  prioritized  requirements.  It  should  be  emphasized  that  these  are 
quite  general,  but  could  be  a  starting  point  from  which  a  specific  set  of  definitized  requirements  could 
ultimately  be  developed,  and  in  turn  a  system  could  be  built. 
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2  Introduction 


2.1  Scope 

This  effort  was  focused  on  performing  research  to  aid  AFRL/HESR  in  identifying  the  basic,  high-level 
requirements  necessary  for  a  follow-on  research  and  development  program  aimed  at  improving  flightline 
diagnostic  capabilities.  The  project  was  initially  aimed  at  identifying  specific  informational  and  visual 
requirements  and  needs  of  the  maintainer  for  a  specific  avionics  diagnostics  task.  Shortly  after  it  began, 
however,  AFRL/HESR  redirected  the  effort  to  emphasize  a  much  broader,  higher-level  approach.  Focus 
shifted  from  a  single  task  approach  to  one  of  documenting  the  basic  flightline  troubleshooting  and 
maintenance  process  and  identifying  the  requirements  that,  if  satisfied,  could  significantly  enhance  the 
maintainers’  efficiency  and  effectiveness  in  turning  aircraft  mission-capable  much  faster  than  is  currently 
possible.  With  the  redirection  of  the  project,  the  initial  predisposition  to  emphasize  a  single  weapon 
system  approach  that  might  later  be  broadened  evolved  into  a  higher-level,  multi-weapon  system  view. 
There  was  little  doubt  that  this  much  broader  look  at  the  problem  would  provide  the  baseline  information 
for  a  follow-on  program  that  would  have  wider  applicability  and  probably  gamer  more  widespread 
support  in  the  maintainer  community  than  any  single  task  or  single  system  approach. 

2.2  Background 

With  the  Aerospace  Expeditionary  Force  (AEF)  concept  now  a  basic  tenet  of  Air  Force  doctrine,  units  are 
expected  to  deploy  far  more  quickly  and  go  to  far  more  austere  locations  than  was  the  case  only  a  few 
years  ago.  In  addition,  those  units  are  expected  to  deploy  with  only  the  basic  maintenance  support 
necessary  to  initiate  operations  upon  arrival.  Given  this  reality,  the  need  for  the  maintainers  who  deploy 
under  these  conditions  to  be  as  capable  and  effective  as  possible  is  taking  on  a  whole  new  significance. 

Despite  the  fact  that  USAF  aircraft  are  typically  more  reliable  and  easier  to  work  on  than  their 
predecessors,  maintainers  are  still  faced  with  considerable  challenges.  Aging  aircraft,  longer  supply  lines, 
Two-level  Maintenance,  declining  manning,  austere  budgets,  consistently  high  operations  tempo,  and  a 
long  list  of  other  challenges  have  combined  in  recent  years  to  make  it  difficult  for  maintainers  to  meet  the 
demands  placed  upon  them.  If  one  assumes  that  these  conditions  are  a  blueprint  for  the  future,  and  every 
indication  suggests  they  are,  then  it  stands  to  reason  that  the  maintainer’s  job  will  only  get  more  difficult 
unless  something  substantive  is  done  to  enhance  their  capability  to  produce  mission-capable  aircraft. 
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The  F/A-22  Program  promises  major  improvements  in  the  ability  of  its  maintained  to  keep  that  aircraft 
flying  and  the  autonomic  logistics  system  for  the  F-35  will  include  additional  capability  enhancements. 
That  is  certainly  good  news,  but  neither  of  those  aircraft  will  be  available  in  numbers  for  a  decade  or 
more.  At  the  same  time,  over  half  of  the  aircraft  that  will  comprise  the  Air  Force  fifteen  years  hence  are 
on  the  flightlines  today.  That  fact  cannot  be  ignored. 

As  will  be  discussed  later  in  this  report,  statistics  clearly  show  that  despite  longer  hours  on  the  flightline 
for  the  maintained  and  some  budget  relief  that  has  improved  the  spare  parts  situation  somewhat,  most 
aircraft  fix  rates  have  not  markedly  improved  in  recent  yead  and  many  have  declined.  Although  this  is 
certainly  a  multi-faceted  problem,  it  is  apparent  that  one  element  of  the  solution  is  that  the  maintainer’s 
ability  to  troubleshoot  and  diagnose  problems  requires  enhancement.  The  Maintenance  Mentor  (MXM) 
research  is  aimed  at  helping  bring  that  about. 
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3  System  Identification 

Because  previous  AFRL/HESR  programs  such  as  Integrated  Maintenance  Information  System  (IMIS)  and 
Predictive  Failures  and  Advanced  Diagnostics  (PFAD)  were  essentially  built  around  the  F-16  and/or  its 
radar,  the  notion  that  the  MXM  research  would  also  be  focused  on  the  F-16  and  its  radar  was  essentially  a 
given.  However,  with  the  broader,  higher-level  look  the  project  had  taken  on  this  approach  had  to  be 
modified.  To  ensure  maximum  flexibility  as  the  project  moved  forward,  the  team  decided  that  parallel 
selections  of  target  weapons  systems  and  subsystems  would  be  conducted.  AFRL/HESR  then  set  out  to 
gather  the  maintenance  data  that  would  be  analyzed  to  identify  potential  weapon  systems  and  subsystems 
upon  which  to  focus  the  research.  Although  most  of  the  necessary  data  for  Air  Combat  Command  (ACC) 
weapons  systems  was  ultimately  acquired,  problems  were  experienced  in  gathering  Air  Mobility 
Command  (AMC)  aircraft  data  that  resulted  in  only  some  basic  information  being  available.  In  the 
interest  of  time,  the  decision  was  made  to  move  forward  with  weapon  system  selections  without  going 
after  additional  information. 

3.1  Weapon  System  Selection 

As  a  baseline  from  which  to  move  forward,  the  team  devised  a  set  of  basic  guidelines  to  provide  some 
structure  for  the  weapon  system  decision  process.  Those  guidelines  are  listed  below: 

♦  High  Maintenance  Man-hour  Consumer 

♦  Probability  of  Data  Availability 

♦  Probability  of  Access  for  Data  Gathering/Knowledge  Acquisition  (KA) 

♦  Potential  for  SPO  Support 

♦  Ease  of  Interfacing  with  the  Aircraft 

♦  Potential  for  Major  Command  (MAJCOM)  Support 

♦  Longevity/Potential  Payback 

♦  Other  Programs  On-going 

No  attempt  was  made  to  prioritize  the  guidelines  and  none  was  necessarily  considered  more  or  less 
important  than  the  others.  Although  that  could  probably  have  been  done  and,  in  addition,  other  guidelines 
might  have  been  devised,  it  was  determined  that  available  research  time  could  be  better  spent  on  other 
facets  of  the  effort. 
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3.1.1  High  Maintenance  Man-hour  Consumer 

From  the  beginning  the  team  decided  that  the  research  would  be  focused  as  much  as  possible  on  weapon 
systems  that  consumed  significant  maintenance  man-hours  because  those  are  the  ones  that  typically 
provide  maintainers  the  most  challenges  and  generate  the  most  headaches  for  both  their  leaders  and  the 
headquarters  for  whom  they  work.  It  was  obvious  that  no  program  would  be  supported  if  it  were  not 
attacking  a  recognized  problem  and  the  team  felt  that  man-hour  consumption  was  a  key  indicator  of 
potential  problems.  A  score  of  1  denotes  a  weapon  system  that  is  not  a  high  maintenance  man-hour 
consumer  while  those  that  were  rated  a  5.  A  weapon  system  was  assessed  to  be  a  high  maintenance 
man-hour  consumer  according  to  command  standards  or  headquarters  input. 

3.1.2  Probability  of  Data  Availability 

Because  the  success  of  any  research  effort  will  be  affected  by  the  ability  of  the  research  team  to  get 
required  data,  a  subjective  assessment  of  how  likely  it  was  that  data  could  be  acquired  was  included  as 
one  of  the  guidelines.  If  data  for  a  weapon  system  could  be  acquired  rather  easily,  that  particular  weapon 
system  would  receive  a  score  of  5.  If  on  the  other  hand  it  was  unlikely  that  die  team  could  get  the  needed 
data,  then  that  weapon  system  was  given  a  score  of  1 .  Scores  between  1  and  5  were  assigned  to  denote 
varying  degrees  of  potential  data  availability. 

3.1.3  Probability  of  Access  for  Data  Gathering/Knowledge  Acquisition 

The  team  then  made  an  assessment  of  how  likely  it  would  be  that  access  would  be  granted  to  units  with  a 
specific  weapon  system  for  data  gathering  and  KA  purposes.  A  score  of  5  meant  that  access  was  highly 
likely,  while  1  meant  it  was  unlikely.  Scores  between  1  and  5  indicated  a  likelihood  of  access  between  the 
two  extremes. 

3.1.4  Potential  for  SPO  Support 

This  was  an  assessment  of  how  supportive  of  a  follow-on  MXM  6.3  effort  a  particular  weapon  system 
program  office  (SPO)  might  be.  SPO  support  could  prove  to  be  vital  to  success  of  any  future  effort  so  the 
team  felt  it  should  be  considered  up  front.  A  high  probability  of  support  rated  a  score  of  5  with  scores 
down  through  1  indicating  less  and  less  likelihood  of  active  support. 
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3.1.5  Ease  of  Interfacing  with  the  Aircraft 

It  was  apparent  to  the  team  that  the  6.3  follow-on  effort  for  which  the  MXM  research  would  be  the 
groundwork  would  require  drawing  information  from  an  aircraft  at  some  point.  The  ease  with  which  this 
could  be  done  was  assessed  by  scores  ranging  from  1  (very  difficult)  to  5  (veiy  easy). 

3.1.6  Potential  for  MAJCOM  Support 

As  in  the  other  guidelines,  this  was  the  team’s  professional  assessment  of  the  likelihood  of  active 
MAJCOM  support  for  the  MXM  research  and  a  follow-on  effort.  High  potential  for  active  support  rated  a 
score  of  5  while  low  potential  was  rated  1,  with  varying  degrees  of  potential  in  between. 

3.1.7  Longevity/Potential  Payback 

If  a  weapon  system  was  assessed  as  likely  to  be  around  for  a  long  time  and  thus  die  payback  from  a  MXM 
6.3  effort  might  be  very  significant,  that  weapon  system  was  rated  a  5.  A  weapon  system  that  was 
assessed  as  probably  not  be  destined  to  be  around  long  and  therefore  bring  little  payback  was  given  a 
score  of  1.  Varying  degrees  of  potential  payback  were  scored  between  1  and  5.  Although  the  team 
realized  that  there  might  not  necessarily  always  be  a  direct  relationship  between  longevity  and  potential 
payback,  it  was  still  felt  that  this  was  something  that  should  be  considered.  It  should  also  be  noted  that  for 
a  follow-on  6.3  effort  the  area  of  potential  payback  will  certainly  be  a  pass  or  fail  criterion  and  will 
require  major  detailed  analysis  to  support  it.  In  that  follow-on  effort  considerably  more  emphasis  in  this 
area  will  be  required  than  was  the  case  here. 

3.1.8  Other  Programs  On-going 

Although  a  numeric  score  was  not  given,  if  the  team  was  aware  of  other  programs  that  might  conflict  with 
the  MXM  effort  or  compete  with  it  or  a  potential  6.3  follow-on  program  for  support  that  fact  was  noted. 

3.1.9  Systems  Not  Considered 

It  should  be  noted  that  some  weapon  systems  were  purposely  not  included  in  the  assessment.  For  a 
number  of  reasons,  including  significant  potential  access  problems,  small  numbers  of  platforms,  and 
security  considerations  to  name  only  a  few,  the  team  decided  not  to  include  low  density,  high  demand 
systems  (i.e.,  F-l  17,  B-2,  Rivet  Joint,  AWACS,  Joint  Stars,  Special  Operations).  In  addition,  pure  training 
systems  such  as  the  T-l,  T-37,  and  T-38  were  not  considered  because  much  of  their  maintenance  is  done 
under  contract  to  commercial  sources.  Given  that  fact,  it  was  decided  that  including  them  in  this  research 
might  present  problems  that  could  cost  undue  amounts  of  time  and  might  generate  other  issues  (i.e.. 
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contractual)  with  which  the  team  was  not  prepared  to  deal.  Likewise,  developmental  systems  such  as  the 
F/A-22,  F-35,  and  CV-22  were  not  considered  in  the  assessment. 


3.1.10  Assessment  Results 

Using  the  aforementioned  guidelines,  the  team  assessed  nine  weapon  systems  (see  Table  1).  Information 
from  various  sources,  including  logistics  data  provided  by  AFRL/HESR,  experience  from  previous 
programs,  inputs  from  MAJCOM  headquarters  contacts,  sources  in  a  number  of  flying  organizations,  and 
inputs  from  other  professional  contacts  were  considered.  The  team  then  gave  each  weapon  system  under 
consideration  a  subjective  score  based  on  the  data  available.  Not  surprisingly,  the  amount  of  available 
information  varied  widely  among  weapon  systems.  Certainly,  more  could  have  been  gathered,  but  it  soon 
became  obvious  that  an  inordinate  amount  of  additional  time  would  have  been  necessary  to  do  so;  thus,  a 
decision  was  made  to  move  ahead. 
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The  F-15,  F-16,  KC-135,  C-130,  and  C-5  were  the  top  five  weapons  systems  based  on  the  scores  attained. 
To  maintain  a  multi-command  flavor  in  the  program  the  research  team  decided  to  attempt  to  focus  on  the 
F-15  and/or  F-16  and  the  C-130  and/or  KC-135.  Not  only  did  both  score  high  in  the  assessment,  but  there 
was  also  the  likelihood  that  both  would  be  in  the  inventory  for  a  long  time.  In  addition,  these  weapon 
systems  represent  both  ends  of  the  technology  spectrum,  which  supports  the  AFRL/HESR  desire  for 
MXM  to  be  applicable  across  as  much  of  the  USAF  fleet  as  possible. 

3.2  Subsystem  Selection 

Although  the  research  team  intended  to  follow  a  similar  process  for  selection  of  a  target  subsystem  as  that 
used  in  weapon  system  selection,  that  proved  difficult  to  do.  Logistics  data  on  ACC  aircraft  was  relatively 
easy  for  AFRL/HESR  to  acquire;  however,  getting  similar  AMC  data  was  a  much  greater  challenge.  As  a 
result,  little  substantive  AMC  data  was  ever  received  so  the  decision  was  made  to  move  forward  using  the 
ACC  data.  A-10,  B-l,  B-52,  F-15,  F-16,  and  EC/HC-130  data  was  used.  Assessments  were  made  of  the 
primaiy  subsystems  listed  below  in  each  of  those  aircraft: 

♦  Fuel 

♦  Hydraulic 

♦  Propulsion 

♦  Landing  Gear 

♦  Flight  Controls 

♦  Radar 

♦  Electronic  Countermeasures 

♦  Electrical 

3.2.1  Selection  Guidelines 

The  team  established  a  set  of  five  guidelines  to  govern  the  assessment  of  subsystems  leading  to  selection 
of  the  target  subsystem  on  which  the  research  would  be  focused.  The  guidelines  are  discussed 
individually  in  the  following  paragraphs. 

3.2.1. 1  Top  Five  Man-hour  Consumer 

The  first  guideline  was  whether  or  not  the  subsystem  in  question  had  consistently  been  one  of  the  top  five 
maintenance  man-hour  consumers  over  the  past  year.  If  the  answer  was  “no,”  the  subsystem  received  a 
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score  of  1 .  If  the  answer  was  “yes”  a  score  of  5  was  awarded.  This  is  the  only  category  in  which  no 
intermediate  scores  were  used  and  it  was  also  the  only  category  in  which  the  score  was  purely  objective. 

3.2.1. 2  Ease  of  Troubleshooting 

A  score  between  1  (Easy)  and  5  (Tough)  was  used  to  gauge  the  challenges  in  troubleshooting  each  of  the 
evaluated  subsystems  on  each  of  the  six  aircraft  types.  The  scores  were  assessed  based  on  a  combination 
of  professional  knowledge  and  experience  of  the  team  and  inputs  gathered  from  discussions  with 
maintainers  in  both  ACC  headquarters  and  field  units.  Although  formal  interviews,  a  survey,  or  some 
other  more  scientific  method  of  gaining  these  inputs  would  have  been  preferred,  this  method  was  chosen 
to  save  time.  As  it  turned  out,  the  inputs  from  these  various  sources  were  quite  consistent  and  agreed  very 
closely  with  information  gained  later  in  the  KA  phase  of  the  research. 

3.2.1. 3  Relevance  to  the  MXM  Project 

The  scores  in  this  category  were  solely  the  professional  opinion  of  the  team  based  on  what  was  known 
about  the  project  and  its  goals  at  the  time.  Scores  were  from  1  to  5,  with  1  meaning  there  was  not  much 
relevance  and  5  meaning  there  was  significant  relevance. 

3.2.1. 4  Potential  Integration  Challenges 

In  this  area,  the  team  attempted  to  assess  the  challenges  that  might  have  to  be  overcome  in  integrating  a 
potential  MXM  solution  into  the  subsystem  and  aircraft  in  question.  Scores  were  assessed  based  on  team 
member  background  and  experience  as  well  as  inputs  gathered  from  conversations  with  SPO 
representatives  and  maintainers  on  the  ACC  staff.  Scores  ranged  from  1  to  5.  If  the  conclusion  was  that 
there  would  potentially  be  huge  challenges,  a  score  of  1  was  awarded.  A  score  of  5  was  given  if  the 
challenges  were  likely  to  be  manageable.  Scores  between  1  and  5  denoted  varying  degrees  of  challenge 
between  the  two  extremes. 

3.2.1. 5  Potential  Improvement  for  Maintainers 

In  this  category,  the  team  attempted  to  assess  the  potential  benefit  to  the  line  maintainers  if  a  MXM 
solution  could  be  leveraged  to  improve  troubleshooting  on  the  aircraft  and  system  in  question.  Scores 
again  ranged  from  1  to  5.  If  little  benefit  would  likely  accrue  to  the  maintainer  from  an  MXM  solution  a 
score  of  1  was  given.  If  major  improvements  for  the  maintainers  were  assessed  to  be  possible  a  score  of  5 
was  given,  with  varying  degrees  of  potential  improvement  between  1  and  5  scored  accordingly. 
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The  results  of  the  assessment  are  shown  in  Table  2  and  the  detailed  data  from  which  those  results  were 
derived  is  listed  in  a  set  of  tables  in  Appendix  J.  Although  some  might  argue  about  the  scores  and  how 
they  were  derived,  the  team  has  since  found  widespread  agreement  with  the  results  among  maintainers 
who  have  seen  them,  specifically  those  interviewed  during  field  visits  and  those  who  participated  in  the 
Decision  Criteria  Development  Workshop  during  this  research  effort. 

Table  2:  Subsystem  Assessment  Results 


Ranking 

Score 

A-10 

B-1 

B-52 

F-15 

F*16 

C-130 

Fuel 

3 

86 

11 

18 

15 

13 

15 

14 

Hydraulic 

8 

51 

7 

10 

9 

9 

7 

9 

Propulsion 

2 

94 

14 

20 

7 

18 

19 

16 

Landing  Gear 

4 

84 

15 

11 

17 

17 

9 

15 

Flight  Controls 

1 

103 

15 

20 

7 

24 

23 

14 

Radar 

7 

59 

9 

10 

7 

12 

11 

10 

ECM 

6 

71 

10 

9 

18 

8 

18 

8 

Electrical 

5 

82 

10 

19 

19 

8 

9 

17 

3.2.2  Target  Subsystem  Selected 

With  the  assessment  results  as  a  baseline,  the  team  set  out  to  determine  the  subsystem  on  which  the 
research  would  be  focused.  Propulsion  was  eliminated  almost  immediately,  primarily  because  there  was 
considerable  work  already  being  done  to  improve  engine  troubleshooting  and  diagnostics  and  MXM 
might  have  been  perceived  to  be  duplicating  that  effort.  The  fuel  system  was  eliminated  because,  although 
there  is  certainly  improvement  needed  in  diagnosing  and  troubleshooting  fuel  problems  on  nearly  every 
aircraft,  AFRL/HESR’s  technical  focus  was  on  an  electronic  system  and  most  fuel  systems  do  not  have  a 
significant  electronic  component  to  them.  Landing  gear  and  electrical  dropped  from  consideration  for 
essentially  the  same  reasons.  Electronic  countermeasures  (ECM)  systems,  with  some  notable  exceptions, 
typically  are  not  especially  difficult  to  troubleshoot,  which  coupled  with  potential  security  concerns 
resulted  in  their  being  dropped  from  consideration.  The  remaining  systems  did  not  rate  high  enough  in  the 
assessment  to  focus  further  effort  on  them. 
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Ultimately,  it  was  decided  that  flight  controls  would  be  the  target  subsystem.  That  decision  was  made  not 
only  because  flight  controls  attained  the  highest  score,  but  for  a  number  of  other  reasons  relating  to  the 
newly  broadened  scope  of  the  effort.  For  one  thing,  most  flight  control  systems  are  not  pure  electronic 
systems  in  that  they  also  have  mechanical  and  hydraulic  elements.  This  afforded  the  opportunity  to 
discern  issues  that  may  have  been  obscured  had  focus  been  only  on  an  avionics  unit.  In  addition,  because 
flight  control  problems  may  derive  from  any  of  the  associated  elements,  this  focus  guaranteed 
examination  of  a  maintenance  task  involving  a  large  diagnostic  search  space  (of  alternatives).  At  the  same 
time  nearly  all  flight  control  systems,  such  as  those  in  the  F-15  and  the  F-16,  have  a  significant  electronic 
component  to  them  and  should  lend  themselves  rather  easily  to  demonstrating  a  diagnostic  or  prognostic 
capability.  This  is  especially  true  in  the  newer  aircraft  such  as  the  C- 17.  In  addition,  most  are  difficult  to 
troubleshoot  and  flight  controls,  even  in  the  newer  aircraft,  tend  to  be  consistently  high  maintenance  man¬ 
hour  consumers.  Furthermore,  flight  control  issues  often  mandate  impoundment  of  an  aircraft.  This  meant 
flight  control  maintenance  was  the  class  of  maintenance  activities  most  likely  to  both  (a)  represent  cases 
dragging  down  aggregate  performance  metrics  (for  repair  time)  and  (b)  involve  the  widest  range  of 
information  assets  and  coordination  with  other  personnel.  Given  those  circumstances,  even  moderate 
improvements  in  troubleshooting  time  could  bring  major  pay  back.  Thus,  it  was  decided  that  flight 
controls  would  be  the  target  subsystem  for  this  research. 
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4  Requirements  Definition 


4.1  Knowledge  Acquisition 

Knowledge  acquisition  (KA)  is  the  label  for  activities  and  procedures  directed  toward  obtaining  task- 
related  knowledge  through  interaction  with  original  sources,  particularly  experts  on  the  topic  at  hand.  This 
section  will  briefly  review  the  background  for  MXM  KA  efforts,  the  tactics  for  pursuing  KA  on  this 
project,  and  the  activities  through  which  the  KA  phase  the  technical  effort  was  conducted.  This  section 
will  review  the  approach  to  gathering  data  on  the  target  maintenance  subject  matter  and  provide  a  brief 
overview  of  the  most  central  outcomes  of  the  KA  effort.  The  subsequent  section  on  cognitive  task 
analysis  (CTA)  will  then  review  other  significant  outcomes  and  conclusions  distilled  from  die  KA 
activities. 

4.1.1  Scope  and  Focus  of  the  KA  and  Subsequent  Efforts 

The  MXM  project  was  initially  focused  on  research  into  the  prospects  and  candidate  means  for  improving 
aircraft/systems  maintenance  activities.  More  specifically,  MXM  was  intended  to  aid  AFRL/HESR  in 
determining  the  feasibility  of  conducting  research  and  development  toward  innovations  reducing  aircraft 
downtime.  The  types  of  innovations  to  be  emphasized  were  those  enhancing  maintainer  capabilities  to 
troubleshoot  system  failures  through  better  diagnostic  aids,  decision  aids,  and  advanced  information 
visualization. 

As  originally  envisioned  and  discussed,  the  MXM  work  was  expected  to  exhibit  the  following  features: 

♦  The  MXM  team  would  focus  on  one  particular  aircraft  and  subsystem  (i.e.,  radar  on  the 
F-16) 

♦  The  team’s  research  efforts  would  concentrate  on  the  cognitive  and  informational  aspects  of 
maintenance  functions  for  the  target  aircraft/system  alone 

♦  The  team’s  research  efforts  within  this  narrow  scope  of  interest  would  lead  to  specific 
recommendations  for  improving  maintenance  of  that  particular  aircraft  and  subsystem. 

As  noted  earlier,  the  MXM  project  was  redirected  soon  after  it  started  and  the  scope  of  research  interest 
was  expanded  to  a  broader,  higher-level  focus.  Thus,  it  was  decided  that  the  MXM  effort  would  be 
framed  with  regard  to  the  following  criteria: 

♦  The  scope  of  system/subsystem  concern  would  be  a  general  class  of  apparatus  applicable  to  a  wide 
variety  of  operational  USAF  aircraft  (discussed  earlier  in  Section  3). 
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♦  This  scope  of  concern  was  agreed  to  be  flight  controls  (see  par.  3.2.2).  A  reasonable  range  of 
operational  USAF  aircraft  would  be  reviewed  with  respect  to  flight  control  maintenance  issues. 

♦  The  MXM  team  agreed  to  achieve  this  breadth  of  range  by  reviewing  both  small  aircraft  (i.e., 
fighters)  and  large  aircraft  (airlift  aircraft  in  this  case). 

4.1.2  Tailoring  the  Knowledge  Acquisition  Itinerary  to  Fit  the  Project 
Scope  and  Focus 

Had  the  research  proceeded  with  the  original  (one  aircrafi/one  subsystem)  scope,  the  KA  itinerary  would 
have  been  tailored  to  emphasize  depth  with  respect  to  the  target  subsystem.  Such  a  depth-orientation 
would  have  involved  detailed  data  gathering  and  interviews  with  a  small  set  of  expert  maintainers 
involved  with  the  target  maintenance  task.  The  revised  project  scope  required  that  data  be  obtained  across 
a  reasonable  sample  of  different  aircraft  and  aircraft  types.  This  in  turn  required  an  adjustment  to  the  KA 
itinerary  to  afford  reasonable  coverage  corresponding  to  the  new  scope  (i.e.,  more  breadth  at  a  lesser 
depth  of  detail).  Various  options  were  reviewed  with  respect  to  travel  costs,  the  range  of  maintenance 
knowledge  that  would  be  required,  and  site  access.  It  should  be  noted  that  site  access  became  a  major 
consideration  because  of  the  exceptionally  high  operational  tempo  due  to  the  continuing  operations  in 
Operation  Enduring  Freedom  (OEF),  the  buildup  for  and  initiation  of  Operation  Iraqi  Freedom  (OIF),  and 
ongoing  Homeland  Defense  commitments.  In  the  end,  it  was  decided  that  the  most  constructive  and 
feasible  course  of  action  would  be  a  two-phase  KA  itinerary: 

♦  Phase  1:  Fighter  Aircraft.  It  was  determined  that  Nellis  AFB,  Nevada  was  a  lucrative  site  for 
obtaining  maintenance  knowledge  on  a  variety  of  fighter  aircraft.  The  range  of  aircraft  stationed  at 
Nellis  included  A-lOs,  F-15Cs,  F-15Es,  and  F-16C/Ds,  as  well  as  die  F/A-22  and  RQ-1.  As  a  training 
and  testing  base,  Nellis  offered  the  prospect  of  operational  conditions  similar  to  those  of  bases  to 
which  access  had  become  problematic  owing  to  ongoing  contingency  operations. 

♦  Phase  2:  Large  Aircraft.  Significant  time  was  spent  considering  options  for  KA  on  a  larger  aircraft. 

As  noted  earlier  in  Section  3.1.10,  a  C-130  or  a  KC-135  was  preferred;  however,  considerable  time 
and  effort  to  arrange  either  of  those  visits  proved  fruitless  due  to  heavy  operational  taskings  in  those 
units.  In  the  end,  it  was  decided  that  going  to  Charleston  AFB,  South  Carolina  to  do  KA  on  the  C- 17 
was  the  best  feasible  option. 

The  actual  itinerary  is  summarized  in  Table  3.  A  four-member  MXM  team  traveled  to  each  of  the 
designated  bases  and  conducted  multiple  days  of  KA  activities. 
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Table  3:  MXM  KA  Itinerary 


SITE 

KA  Team  Members 

Days  on-site 

Aircraft  Reviewed 

Nellis  AFB,  NV 

♦  John  Jacobs  (NGIT) 

♦  Randall  Whitaker  (NGIT) 

♦  Capt.  David  Lemery  (AFRL) 

♦  Capt.  Brian  Tidball  (AFRL) 

15 -18  April 

2003 

♦  A-10 

♦  F-15C 

♦  F-15E 

♦  F-16C/D 

♦  F/A-22 

♦  RQ-1  (Predator 
UAV) 

Charleston  AFB,  SC 

♦  John  Jacobs  (NGIT) 

♦  Randall  Whitaker  (NGIT) 

♦  Capt.  David  Lemery  (AFRL) 

♦  Capt.  Brian  Tidball  (AFRL) 

29  April  - 1  May 

2003 

C-17 

4.1.3  Tailoring  the  Knowledge  Acquisition  Tactics  to  Fit  the  Project 
Scope  and  Focus 

4.1. 3.1  Setting  KA  Criteria  Appropriate  to  the  Revised  Plan 

Shifting  the  scope  of  the  MXM  effort  also  meant  that  adaptations  had  to  be  made  to  the  KA  tactics  to  be 
employed.  The  most  commonly  used  on-site  KA  tactics  include  structured  interviews  with  subject  matter 
experts  (SMEs),  observations  of  target  tasks  being  accomplished,  and  task  simulations.  The  shift  to  a 
more  general  scope  induced  several  new  or  newly  prioritized  criteria  for  choosing  the  specific  KA  tactics. 
Some  of  the  most  important  such  criteria  were: 

♦  Minimizing  time  consumption  on-site.  The  original  allocations  for  KA  travel  time  and  funding  were 
based  on  a  narrow  study  of  one  subsystem  of  one  aircraft.  Detailed  data  collection  (i.e.,  observations) 
for  representative  maintenance  activities  on  the  seven  different  aircraft  being  covered  would  have 
potentially  required  more  time  than  was  available.  Thus,  KA  tactics  were  tailored  against  available 
time. 

♦  The  trade-offs  between  depth  and  breadth  in  data  collection.  The  available  time  would  permit  drill 
down  only  so  deep  on  each  of  the  subject  aircraft.  This  meant  a  decision  had  to  be  made  on  how  many 
relevant  dimensions  of  the  flight  control  maintenance  activities  would  be  explored  and  the  depth  to 
which  each  would  be  explored.  This  became  an  involved  set  of  trade-off  evaluations. 
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♦  Minimizing  collection  of  data  too  fine-grained  to  be  informative  on  the  stated  topic.  Details  of  work 
procedures  might  be  peculiar  to  one  or  more  of  the  subject  aircraft,  resulting  in  the  collection  of  fine¬ 
grained  data  of  little  applicability  at  the  more  general  level  of  interest  that  was  now  the  focus. 
Similarly,  details  of  those  tools  and  aids  peculiar  to  one  or  another  aircraft  would  be  unlikely  to 
illuminate  issues  and  problems  pertinent  to  all  the  aircraft. 

♦  Maximizing  the  generalizability  of  task  data  and  derived  models  with  respect  to  as  many  aircraft  as 
possible.  To  deal  with  the  expanded  scope  of  concern,  target  data  and  knowledge  of  broad 
applicability  to  flight  control  maintenance  functions  across  all  aircraft  was  needed.  This  meant 
tailoring  of  the  KA  plan  would  be  required  to  concentrate  on  maintenance  processes  and  procedures 
of  sufficient  generality  to  be  relevant  across  all  the  subject  aircraft. 

♦  Maximizing  the  generalizability  of  data  on  the  tools  and  aids  employed  in  flight  control  maintenance. 
The  MXM  effort  was  geared  toward  exploring  the  role  of  current  and  prospective  maintenance 
support  tools.  However,  some  of  the  support  aids  (specifically  diagnostic  decision  support  aids)  are 
specific  to  only  one  or  another  of  the  subject  aircraft.  It  was  decided  to  emphasize  data  collection  on 
those  tools  most  generally  applicable  across  all  aircraft.  This  is  why  special  attention  was  given  to 
examining  the  tool  inventories  of  the  units  visited. 

♦  Administrative  constraints.  Given  the  revised  and  more  general  scope  of  MXM  efforts,  consideration 
was  given  to  how  to  supplement  the  on-site  KA  with  other,  wider-ranging  forms  of  data  collection. 
The  use  of  a  written  questionnaire  or  survey  was  one  of  the  alternative  tactics  examined  in  most 
detail.  As  it  turned  out,  there  are  significant  USAF  administrative  constraints  on  broadcasting  such 
instruments  to  operational  units.  The  overhead  for  overcoming  these  administrative  constraints,  added 
to  the  practical  overhead  costs  of  conducting  such  a  survey,  resulted  in  the  decision  that  this  tactic 
was  not  feasible  within  the  planned  timeframe. 

4.1. 3.2  Selecting  KA  Tactics  Appropriate  to  the  Revised  Plan  and  Criteria 

Once  revised  criteria  for  the  KA  approach  were  established,  KA  tactics  (themes,  questions,  instruments) 

were  developed  that  reflected  them.  The  most  important  KA  tactics  devised  for  the  MXM  effort  included 

the  following: 

♦  A  preliminary  KA  plan  to  serve  as  a  structured  guide  for  the  KA  team.  A  1 0-page  KA  plan  was 
drafted  and  distributed  to  the  team.  This  document  outlined  the  rationale  for  KA  tactics  to  be 
employed  and  provided  a  notional  set  of  key  questions  and  issues  to  serve  as  a  guide  during  the  KA 
sessions.  This  plan  (see  Appendix  C)  provided  die  Irasie  pic-visit  preparation  for  die  KA  team. 

♦  Emphasis  on  a  proactive  “straw  man  ’’  review  rather  than  passive  elicitation.  The  MXM  team  had 
access  to  substantial  information  on  general  maintenance  issues  and  facts.  Based  on  this  information 
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base,  the  KA  team  was  in  a  position  to  get  a  head  start  on  outlining  some  general  points  to  put  in  front 
of  the  SMEs  (as  opposed  to  starting  from  scratch  and  taking  time  to  draw  out  such  basics  during  the 
interviews).  Much  time  can  be  conserved  if  SMEs  are  presented  with  an  initial  “straw  man”  (model, 
diagram)  to  which  they  can  immediately  react.  In  this  case,  the  “cascade”  approach  to  mapping  the 
flight  control  maintenance  process  was  aided  by  presenting  maintenance  SMEs  with  a  set  of 
presumed  stages  in  the  maintenance  process  and  then  asking  them  to  arrange  these  elements  into  a 
representative  model  of  that  process  path. 

♦  Surveying  the  tool  inventories  of  frontline  maintenance  units.  A  point  was  made  to  request  time  and 
access  to  review  unit  tools  with  a  focus  on  those  specifically  employed  in  flight  control  maintenance. 
In  preparation  for  the  KA  trips  a  structured  questionnaire  form  (see  Appendix  E)  was  developed  for 
documenting  information  about  flight  control-related  tools  and  instruments. 

♦  Interviews  with  support  and  other  relevant  personnel.  Flightline  maintained  are  supported  by  a 
number  of  people  and  activities  in  the  course  of  their  work.  Support  personnel,  such  as  Air  Force 
Engineering  and  Technical  Services  (AFETS)  and  contractor  technical  support  staff,  and  support 
functions  (i.e.,  on-base  component  test  station  shops)  were  visited  wherever  possible.  This  allowed  a 
broader  perspective  on  maintenance  operations  as  actually  practiced,  and  it  afforded  the  team 
additional  insight  regarding  the  means  by  which  the  frontline  maintained  reach  back  for  information 
as  necessary. 

♦  Collection  of  additional  documentary  data  as  available.  In  addition  to  the  interview  sessions  and  tool 
questionnaires,  a  point  was  made  to  solicit  other  available  data  such  as: 

^  Overall  maintenance  operations  statistics  (for  the  unit) 

>  More  specific  maintenance  functional  statistics  (overall,  and  especially  for  flight  control 
issues) 

>  Incidence  statistics  for  flight  control  problems  (e.g.,  how  often,  how  long  it  takes) 

>  Data  on  specific  tools  (especially  those  employing  automation,  programming,  and/or  resident 
software) 

>  Inventory  of  information  resources  available  to  the  maintenance  team 

^  Data  on  the  maintenance  team’s  supply  chain  and  supply  procedures 

>  Data  on  any  recent  or  pending  changes/innovations  in  flight  control  maintenance 

>  Pointed  and/or  data  on  better  tools  identified  by  the  SMEs  in  the  coude  of  the  interviews 

4.1. 3.3  “Cascade”  Approach  to  Progressive  Knowledge  Elicitation 

Given  the  scope  and  timeframe  for  the  KA  activities,  it  would  be  necessary  to  obtain  maximally 

generalizable  data  (facts,  models)  in  minimum  time.  The  final  KA  plan  implemented  a  four  phase 
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“cascade”  approach  to  exploring  maintenance  process  and  associated  information  utilization.  A  core 
model  of  the  maintenance  process  would  be  elicited  and  then  employed  as  the  common  reference 
framework  for  collating  details  on  critical  dimensions  of  the  maintenance  task.  As  implemented,  this 
cascade  tactic  consisted  of  the  following  components: 

♦  First,  a  general  model  of  the  maintenance  process  flow  (from  problem  notification  through  to 
returning  the  aircraft  to  duty)  would  be  elicited,  refined,  and  validated  with  subsequent  SMEs. 

♦  Second,  this  model  would  be  used  as  the  focal  framework  for  eliciting  comments  on  procedures, 
practices,  and  problems  related  to  each. 

♦  Third,  this  model  (and  the  data  on  procedural  matters)  would  be  used  as  the  framework  for  polling  the 
SMEs  on  what  data  or  information  they  needed  or  expected  to  employ  at  each  stage  of  the  overall 
work  process  and  with  regard  to  the  procedural  issues  developed  in  the  second  stage. 

♦  As  the  fourth  phase  of  this  exploration,  the  team  visited  selected  support  sections  to  review  flight 
control  tools  and  instruments  and  to  gather  data  on  those  tools  and  their  usage. 


This  four  phase  KA  progression  is  laid  out  in  Table  4.  As  the  table  indicates,  the  team  was  able  to  rapidly 
achieve  consensus  results  on  the  goals  of  the  first  three  phases,  and  reviewed  flight 

Table  4:  Four  Phase  KA  Strategy 


PHASE  1 

PHASE  2 

PHASE  3 

PHASE  4 

Establish  basic 
maintenance  process 
flow 

Correlate  key  info 
requirements 

Correlate  primary 
t  data/info  resources 

Correlate  tools 
ind  instruments 

Interview  SMEs  to  derive 
the  main  steps  and 
progression  of  typical 
process  path 

Probe  SMEs  on  key 
issues  and  questions  for 
each  of  the  steps  in  the 
process  path 

Poll  SMEs  on  identity  and 
utility  of  data  and 
information  employed  on 
these  issues/questions  in 
the  process  path 

♦  Probe  for  what  the 
SMEs  employ 
during  the  process 
path 

♦  Probe  for  issues 
and  problems 

♦  Solicit 
suggestions/ 
recommendations 

♦  Inventory  support 
shop  tool  inventory 
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BEST  AVAILABLE  COPY 


PHASE  1 

PHASE  2 

PHASE  3 

PHASE  4 

Establish  basic 
maintenance  process 
flow 

Correlate  key  info 
requirements 

Correlate  primary 
data/info  resources 

Correlate  tools 
and  instruments 

♦  Begun  in  session  1, 

Day  1 

♦  Consensus  path  model 
achieved  in  first 
session 

♦  Model  validated  in  all 
subsequent  sessions 

♦  Structured  probing 
began  in  second 
session,  Day  1 

♦  Key  issues/questions 
carried  over  and 
presented  for 
subsequent  groups’ 
review 

♦  Stable/consensus  set 
of  questions  and 
issues  discernible  by 
end  of  Day  2 

♦  General  questioning 
on  info  assets  began 
with  session  1 ,  Day  1 

♦  Structured  probing 
began  halfway  through 
Day  1 

♦  Specifics  of  info 
resources  vary  among 
A/C  types 

♦  Attention  paid  to 
human  info  assets 
(e.g.,  tech  support) 

♦  Obtained  data  on 
diagnostic  and 
reference  aids  in 
interviews 

♦  Probed  on  usage 
of  tools  and  aids 

♦  Visited  support 
sections  to 
inventory  flight 
control  tools  (F- 
15C,  F-15E,  F-16, 
C-17) 

control  maintenance  tools  on  four  of  the  five  aircraft  types  for  which  a  dedicated  support  section  was  in 
operation. 


4.1.4  Conducting  Knowledge  Acquisition  On-site 

At  both  sites,  a  meeting  room  was  made  available  for  the  team’s  use,  and  all  the  SME  interviews  were 
conducted  there.  The  four  members  of  the  KA  team  (Lemeiy,  Tidball,  Jacobs,  and  Whitaker)  were  present 
in  all  the  interviews.  The  sessions  were  typically  scheduled  to  permit  at  least  90  minutes  with  each  set  of 
SMEs.  Auxiliary  visits  to  on-base  sites  (i.e.,  support  sections  and  test  stations)  were  made  as  necessary  In 
addition  to  the  interview  sessions,  Capt.  Lemeiy  and  Capt.  Tidball  solicited  documentary  evidence  (e.g., 
statistical  summaries  and  maintenance  reports)  relating  to  both  flight  control  maintenance  in  particular 
and  maintenance  performance  overall. 


All  sessions  were  conducted  in  accordance  with  the  agenda  established  by  the  on-site  hosts.  There  was 
only  one  session  (at  Nellis)  where  a  miscommunication  resulted  in  the  designated  set  of  SMEs  not  being 
available  for  the  scheduled  interview.  In  this  case,  the  SMEs  were  re-scheduled  to  participate  in  a  later 
session  on  the  same  aircraft  (F-15),  so  their  inputs  were  not  lost.  Some  on-site  adjustments  were  made 
during  the  visits,  allowing  the  team  to  obtain  data  and  interview  SMEs  beyond  what  was  afforded  by  the 
original  agenda.  For  example,  on-site  adjustments  at  Nellis  allowed  data  to  be  gathered  on  maintenance 
issues  for  the  RQ-1  Predator  UAV  and  the  newly  arrived  F/A-22  Raptor. 
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Each  of  the  interview  sessions  began  with  an  introduction  of  the  team,  an  overview  of  the  MXM  project, 
and  the  purpose  of  the  session.  All  SMEs  were  asked  to  provide  basic  identification  information  on  a 
sign-in  sheet  (Appendix  D)  and  were  advised  of  the  team’s  non-attribution  policy  regarding  the 
information  they  would  be  providing.  A  lead  interviewer  facilitated  each  session  with  the  other  team 
members  participating  as  circumstances  warranted. 

The  set  of  key  issues  assembled  prior  to  the  KA  visits  (Appendix  C)  was  used  as  the  basis  for  the 
interviews.  As  time  went  on,  the  focal  points  arising  in  the  early  interviews  were  fed  forward  as  talking 
points  for  subsequent  sessions.  Unexpected  significant  claims  and  answers  arising  in  each  session  were 
specifically  noted.  A  good  example  of  this  occurred  in  the  very  first  session  at  Nellis,  where  the  SMEs 
indicated  that  a  near-majority  of  flight  control  problems  ended  up  being  a  matter  of  “switchology”  (i.e., 
incorrect  switch,  knob,  or  control  settings).  This  issue  was  flagged  and  care  was  taken  to  question  SMEs 
in  all  subsequent  sessions  about  it.  In  addition  to  the  switchology  issue,  examples  of  other  such  “pop  up” 
priority  questions  included: 

♦  Quality  (e.g.,  clarity,  completeness,  informativeness)  of  aircrew  problem  reports  (AFTO  Form  781 A 
write-ups)  and  the  procedures  for  such  reporting 

♦  What  one  type  of  information  the  maintenance  team  would  most  like  to  obtain  at  the  outset  of  the 
process  path 

♦  Maintainer  descriptions  of  the  indicators  of  maintenance  expertise 

♦  Distinctions  and  differentiations  between  the  knowledge  and  skills  evidenced  by  expert/experienced 
maintained  and  novice/newer  ones 

♦  Distinctions  between  the  focus  for  flight  control  maintenance  training  and  what  actually  happens  out 
on  the  flightline 

♦  Problems  and  deficiencies  deriving  not  from  LRUs  but  from  what  interconnects  LRUs  (e.g.,  wiring, 
connectors) 

♦  Tradeoffs  between  paper  and  electronic  decision  aids 

♦  The  issues  surrounding  efficiency  in  setting  up  and  working  within  an  impound  team 

♦  Personally  or  locally  developed  information  resources  and  aids 

♦  Which  categories  of  tools  (or  components  required  to  employ  said  tools)  ended  up  costing  time  and 
effort 

♦  Coordination  issues  in  establishing  and  operating  an  impound  team 

♦  Coordination  issues  in  cycling  LRUs  back  upstream  (i.e.,  to  test  stations  and  beyond) 

♦  Coordination  issues  in  finally  certifying  a  maintenance  process  as  completed 
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♦  Degree  of  reliance  on  automated  aids  (i.e.,  portable  maintenance  aids  (PMAs) 

♦  Degree  of  reliance  on  local  technical  support  staff  (e.g.,  AFETS,  contractors) 

No  video  or  audio  recording  was  performed  in  the  KA  sessions.  The  products  from  the  KA  sessions 
consisted  of  individual  sets  of  handwritten  notes,  transcriptions  of  material  written  on  whiteboards  during 
sessions,  flip  chart  sheets  (where  used),  SME-supplied  photocopies  and  tool  data,  some  FI  cards,  and  data 
requested  from  on-base  QA  and  other  maintenance  staff  offices.  The  volume  of  data  from  the  KA  session 
was  very  large.  Owing  to  time  and  budget  constraints,  transcription  of  the  KA  session  notes  was  limited 
to  those  from  the  Nellis  sessions. 

4.1.5  Laying  out  the  Course  of  a  Representative  Flight  Control 
Maintenance  Process  Path 

The  very  first  session  at  Nellis  began  with  a  review  of  a  straw  man  sequence  of  steps  for  a  typical  or 
representative  flight  control  maintenance  process  generated  prior  to  the  KA  trip.  It  included  a  series  of  pro 
forma  steps  with  attention  to  (a)  repair  actions,  (b)  administrative  coordination,  and  (c)  decision  processes 
relevant  to  the  maintenance  cycle.  The  candidate  steps  were  drawn  from  preliminary  readings  on  the 
maintenance  process  and  professional  experience  of  the  research  team  members.  A  set  of  Post-It  notes 
(one  labeled  for  each  step)  was  laid  out  on  the  interview  room  conference  table.  The  steps  represented  by 
each  Post-It  note  were  given  brief  explanations.  The  initial  sequence  shown  to  the  SMEs  was  as  follows: 

♦  Problem  Identification  (acknowledge  and  describe  a  flight  control  issue) 

♦  Problem  Reporting/Documentation  (communicating  and  recording  the  fact  and  the  type  of  flight 
control  issue;  “get  the  case  started”) 

♦  Unit  Coordination1  to  get  the  aircraft  into  the  maintenance  cycle  (make  arrangements  for  maintenance 
work;  convene  an  impound  team) 

♦  Maintenance  Setup/Preparation  (park  the  aircraft;  assemble  tools) 

♦  Diagnosis  (probe  to  see  what  the  problem  is) 

♦  Prognosis  (decide  prospects  for  solutions) 

♦  Repair  Actions/Activities  (die  actual  work  of  fixing  the  plane) 

♦  Supply/Requisition  (in  parallel  to  everything  else;  actually  outside  the  main  sequence) 

♦  Completion  Decision  (decide  that  we’re  finished) 

♦  Test/Evaluate  the  Solution/Repairs  (make  sure  it  now  works  right) 


1  The  SMEs  were  advised  that  the  team  used  die  term  “Unit  Coordination”  to  mean  “coordinating  actions  and  responsibilities  among  relevant 
players." 
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♦  Certification/V alidation  of  Solution  (sign  off  that  the  plane  is  now  fixed) 

♦  Solution  Reporting/Documentation  (paperwork;  records) 

♦  Maintenance  Stand-Down  (clean  up;  put  away  tools,  etc.) 

♦  Unit  Coordination  to  get  the  Aircraft  Back  into  Service 

SMEs  were  then  asked  for  their  opinions  on  this  straw  man  layout  of  a  typical  or  representative 

maintenance  process  path.  They  were  invited  to  make  any  comments  or  corrections  they  thought  would 

make  it  more  accurate.  The  first  set  of  SMEs  made  several  comments  on  this  initial  sequence,  including: 

♦  The  Completion  Decision  step  is  accomplished  after,  and  not  before,  the  Test/Evaluation  and  the 
Certification/V  alidation  steps. 

♦  It  is  difficult  to  separate  the  two  first  steps  (Problem  Identification  and  Problem 
Reporting/Documentation)  in  all  cases.  It  is  often  the  case  that  identification  of  the  problem  and 
reporting/documentation  occur  together  (e.g.,  during  the  pilot  debriefing). 

♦  There  is  often  some  measure  of  Prognosis  occurring  up  front  with  the  Problem  Identification  and 
Reporting/Documentation  steps.  In  other  words,  experienced  maintainers  often  have  a  sense  of 
maintenance  prospects  (roughly  how  much  time;  how  big  a  job)  at  the  point  they  received  the  initial 
problem  description  (e.g.,  from  the  pilot). 

♦  The  Test/Evaluation  and  the  Certification/V  alidation  steps  are  most  commonly  blended  together  in 
practice.  Although  distinct,  these  two  functions  are  typically  interleaved  in  the  course  of  flight  control 
maintenance. 

♦  The  Diagnosis  and  Maintenance  Setup/Preparation  steps  do  not  always  occur  independently  and/or  in 
the  order  originally  presented.  Sometimes  Diagnosis  is  pretty  much  done  up  front  (e.g.,  for  an  obvious 
fault),  and  the  Setup/Preparation  is  done  with  knowledge  of  what  the  problem  is.  Sometimes 
Diagnosis  and  maintenance  Setup/Preparation  go  hand-in-hand,  as  when  something  uncovered  during 
Diagnosis  requires  them  to  back  up  to  do  more  Setup/Preparation  before  proceeding. 

♦  Depending  on  circumstances,  the  course  of  an  actual  flight  control  maintenance  process  can  be 
cyclical  or  iterative  (as  contrasted  with  the  basic  linear  schema  presented  to  the  SMEs).  In  other 
words,  it  is  not  always  accurate  to  characterize  the  maintenance  process  as  a  one-pass  stepwise 
progression  through  the  given  steps. 

♦  One  exception  to  the  linear  stepwise  progression  occurs  when  it  is  possible  and/or  appropriate  to 
“skip”  or  “jump”  along  to  a  later  step  without  having  to  stop  and  do  an  intermediate  one.  An  example 
of  this  would  be  when  the  solution  is  immediately  apparent  and/or  readily  implementable. 
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♦  A  more  common  exception  to  the  linear  stepwise  progression  relates  to  the  “troubleshooting  cycle.” 
An  actual  maintenance  process  often  consists  of  multiple  iterations  through  the  section  of  the 
sequence  from  Diagnosis  through  Certification/Validation.  This  sort  of  cyclical  process  path  is  most 
evident  when  swap-and-test  tactics  are  employed  to  address  the  flight  control  problem  (i.e.,  change  a 
component  -  test  if  that  fixes  the  problem  -  repeat  as  necessary). 

Once  this  review  and  modification  cycle  was  completed,  the  SMEs  agreed  that  the  revised  sequence  was  a 
good  representative  model.  This  revised  model,  illustrated  in  Figure  1,  was  subsequently  reviewed  and 
validated  by  all  the  SME  groups  interviewed  at  both  Nellis  and  Charleston. 


Possible 

'Leaps' 


f S  Problem  Identification  ^ 
Problem  Reporting  /  Documentation 
Unit  Coordination  (A/C  ->  MX) 

MX  Setup  /  Preparation 
f  ■  ^  Diagnosis 
^  Prognosis 
Repair  Actions  /  Activities 
Test /  Evaluate  Splutipir  / Repairs 
Certify  /  Validate  Solution 
Compietion-Daciskm- 
Solution  Reporting  /  Documentation 
MX  Stand-Down  /  Clean-Up 
Unit  Coordination  (A/C  ->  Duty) 


Supply  / 
Requisition 


T 


A 


Troubleshooting! 


Figure  1 :  Consensus  Layout  for  a  Representative  Maintenance  Process  Path 
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Figure  1  uses  shaded  boxes  to  group  together  steps,  which  the  SMEs  indicated  might  be  blurred  or 
interleaved  together.  There  were  three  such  “clusters”  noted  by  the  SMEs: 

♦  Problem  Identification  -  Problem  Reporting/Documentation 

♦  Maintenance  Setup/Preparation  -  Diagnosis 

♦  Test/Evaluate  -  Certify/Validate 

A  set  of  arrows  indicate  the  “possible  leaps”  (forward  through  the  process  path)  the  SMEs  indicated  might 
occur.  Two  such  “leaps”  were  consistently  mentioned  by  the  SMEs.  These  occur  when  information 
obtained  during  the  Problem  Identification/Reporting  “cluster”  leads  immediately  to  a  result  (perhaps 
only  a  tentative  result)  for  Diagnosis  and/or  Prognosis.  The  SMEs  in  multiple  sessions  at  both  bases  noted 
information  obtained  at  the  point  of  initial  reporting  might  “leap  forward”  to  an  immediate  repair  action. 
Insofar  as  this  presumes  a  hypothesis  on  the  nature  of  the  problem  (i.e.,  a  diagnosis),  this  type  of  “leap”  is 
not  separately  illustrated  in  Figure  1. 

A  shaded  arrow  on  the  right  indicates  die  region  of  the  process  path  during  which  supply/requisition 
activities  may  be  proceeding  in  parallel.  No  SME  in  any  session  characterized  supply/requisition  as 
anything  other  than  a  parallel  activity  occurring  outside  the  primary  maintenance  process  path.  In 
addition,  another  set  of  arrows  is  used  to  delimit  the  range  of  process  steps  incorporated  into  the  iterative 
“troubleshooting  cycle”  which  the  SMEs  noted  as  a  common  feature  of  the  process.  These  are  the  steps 
starting  with  Diagnosis  and  continuing  through  the  Test/Evaluate  -  Certify/Validate  clusters.  Multiple 
SMEs  in  multiple  sessions  noted  this  cycle  as  being  particularly  apparent  when  pursuing  a  “swap-and- 
test”  approach  to  fixing  a  flight  control  problem. 

The  process  model  outlined  in  Figure  1  coalesced  into  near-final  form  on  the  first  day  of  the  first  KA  visit 
at  Nellis.  The  model  was  subsequently  presented  to  every  other  maintenance  group  interviewed  at  both 
Nellis  and  at  Charleston.  No  later  group  had  any  significant  comments  or  recommendations  for  improving 
the  model  at  the  given  level  of  generality.  As  such,  the  model  was  effectively  validated  with  the  first 
session  on  the  first  day,  which  allowed  it  to  be  leveraged  for  the  other  KA  goals  at  a  very  early  point. 

Allowing  for  the  “clustering”  noted  or  suggested  by  the  SMEs,  the  final  process  path  representation 
breaks  out  into  eight  primary  steps  or  phases  as  follows: 

♦  Problem  identification/reporting 

♦  Front-end  unit  coordination  (to  get  the  aircraft  into  the  maintenance  process) 

♦  Maintenance  setup/preparation 
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♦  Troubleshooting  cycle  (subsuming  testing,  diagnosis,  prognosis,  repair  actions,  and  testing/evaluation 
to  the  point  the  work  is  deemed  complete) 

♦  Solution  reporting/documentation 

♦  Solution  validation/verification  and  process  completion  decision 

♦  Maintenance  stand-down/cleanup 

♦  Back-end  unit  coordination  (to  get  the  aircraft  back  to  duty) 

There  were  multiple  reasons  for  settling  on  this  eight-way  breakout.  First,  it  reflects  the  points  of 
transition  or  natural  boundary  events  evident  in  the  SMEs  comments  about  the  progress  of  their 
maintenance  work.  Second,  this  decomposition  matches  points  of  transition  for  the  changing  set  of 
specific  individuals  working  on  the  problem  at  any  given  time.  Third,  this  decomposition  is  the  one  most 
consistently  applicable  across  all  the  SME  groups  interviewed.  This  eight-step  decomposition  will  be 
referred  to  during  the  remainder  of  this  report. 

4.1.6  Identifying  Information  Requirements  for  Typical  Flight  Control 
Maintenance  Process  Path 

The  rapid  delineation  of  a  consensus  model  for  the  typical  maintenance  process  path  allowed  the  team  to 
start  associating  information  requirements  with  process  path  steps  early  in  the  KA  effort.  The  approach 
was  to  step  the  SMEs  through  the  acknowledged  process  path  map  and  probe  at  each  step  for  the  key 
issues  they  needed  to  resolve  to  accomplish  that  step.  In  some  cases,  this  boiled  down  to  a  set  of  key 
questions  to  be  asked  and  answered.  In  some  other  cases,  this  yielded  a  set  of  key  issues  needing  to  be 
addressed  (and  which  were  not  readily  translated  into  key  questions  per  se).  The  following  is  a  summary 
listing  of  the  most  commonly  cited  information  requirements  for  the  flight  control  maintenance  process 
path. 


4.1. 6.1  Problem  Identification/Reporting 

♦  What  is  the  nature  of  the  flight  control  malfunction? 

♦  What  are  the  malfunction’s  symptoms? 

♦  When  (at  what  points)  during  the  flight)  did  these  symptoms  occur? 

♦  What  was  the  pilot  doing  at  the  time(s)  the  malfunction  was  apparent? 

♦  What  questions  do  the  fault  tree  (diagnostic  guide)  recommend  or  require  once  we  start  tracing  the 
problem  using  that  aid? 

♦  How  did/does  the  malfunction  affect  the  mission? 
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♦  In  what  axis  or  axes  was  the  flight  control  problem  observed  or  noted? 

♦  What  other  indications  can  be  cited  (e.g.,  by  the  pilot)  to  help  describe  the  nature  of  the  apparent 
malfunction? 

♦  What  other  indications  can  be  cited  (e.g.,  by  the  pilot)  to  help  describe  the  operational  context  or 
flight  conditions  at  the  time  the  apparent  malfunction  was  observed/noted? 

♦  Are  there  any  physical  attributes  or  features  of  the  plane  that  would  seem  to  correlate  with  the  stated 
problem? 

♦  Are  there  any  configurations  of  the  plane  or  its  controls  that  seem  to  correlate  with  the  nature  of  the 
stated  problem? 

♦  Does  the  reported  fault  appear  to  be  intrinsic  to  the  aircraft’s  systems,  or  was  it  induced  (by  pilot 
actions)? 

♦  Were  there  any  visual  observations  of  the  plane  in  flight  that  help  to  contextualize  the  occurrence  of 
the  apparent  malfunction? 

♦  What  were  the  flight  parameters  at  the  time(s)  the  apparent  flight  control  malfunction  was 
observed/noted  (e.g.,  altitude,  AOA,  etc.)? 

♦  What  was  the  pilot  doing  (or  trying  to  do)  at  the  time  he/she  perceived  an  apparent  flight  control 
malfunction?2 

♦  Did  the  apparent  malfunction  occur  only  once,  or  did  it  re-occur  during  the  flight? 

♦  Was  there  uncommanded  flight  control  movement  observed?3 

♦  What  was  the  aircraft  configuration  during  the  flight/mission  during  which  the  apparent  flight  control 
malfunction  was  observed?4 

♦  What,  if  any,  system  or  systems  are  obviously  offline? 

♦  Is  the  apparent  problem  obviously  a  procedural  or  configuration  glitch  resulting  from  (e.g.): 

■  Improper  sequencing  of  actions  (i.e.,  error  in  procedure) 

■  Errors  or  problems  with  controls/switch  settings  (i.e.,  switchology) 

■  Transient  control  or  (subsystem  state  which  can  be  corrected  simply  by  doing  a  “reset”) 

♦  What  fault  codes  are  associated  with  the  apparent  problem? 

♦  Does  a  reset  resolve  the  problem  (to  the  extent  it  can  be  observed/replicated  on  the  ground)?5 

2  In  contrast  with  the  previous  point,  this  refers  to  the  mission-  or  flight-related  tactics,  maneuvers,  etc.,  at  the  time  the  apparent  fault  occurred. 

3  Uncommanded  flight  control  movement  was  repeatedly  cited  by  multiple  SMEs  as  a  very  serious  indicator  in  flight  control  problems.  Several 
SMEs  (across  multiple  sessions)  cited  uncontrolled  movement  as  a  factor  which  would  always  ensure  an  impound. 

4  This  point  was  explained  to  be  focused  on  mission-related  aircraft  states  (e.g.,  weapons  load)  which  might  contribute  to  explaining  aberrant 
flight  control  behaviors.  Examples  given  in  this  session  were  fuel  imbalance  and  any  other  weight  distribution  feature  which  might  have  affected 
the  aircraft  center  of  gravity  at  the  time  of  problem  occurrence. 

5  This  is  not  exactly  redundant  with  the  preceding  allusion  to  “reset”  in  this  list.  In  the  first  case,  the  SMEs  were  referring  to  an  initial  or  up-front 
quick  resolution  because  something  specifically  cued  them  the  resolution  was  a  matter  of  a  reset.  In  this  case,  die  SMEs  were  indicating  a  step  or 
point  at  which  a  reset  was  attempted,  even  though  they  had  not  previously  identified  the  apparent  problem  as  a  “quick  fix  via  reset.” 
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♦  What  specific  information/clues  can  the  pilot  provide  that  can  be  documented  in  writing  up  the 
problem  report? 

♦  How  much  relevant  information  can  we  get  from  the  pilot  at  the  point  of  his/her  initial  report? 

♦  Are  the  forms  (and/or  other  documentation  of  this  problem)  accurately  filled  out? 

♦  What  MFL  code(s)  can  be  identified  for  this  problem? 

♦  What  BIT  codes  or  other  onboard  data  can  be  immediately  obtained  to  shed  more  light  on  this 
reported  problem? 

♦  What  candidate  diagnoses  suggest  themselves  at  this  point? 

4.1. 6.2  Unit  Coordination 

The  unit  coordination  (to  get  the  aircraft  into  the  maintenance  cycle)  include: 

♦  What  maintainers  need  to  be  assigned  to  work  this  problem? 

♦  What  place/space  will  be  used  to  work  this  problem? 

♦  Do  we  have  to  arrange  for  hangar  time? 

♦  What  are  the  pending  weather  conditions?  (i.e.,  can  we  work  the  problem  outside?) 

♦  When  is  the  aircraft  scheduled  to  fly  next? 

♦  What  do  we  tell  the  Pro  Super  about  the  aircraft’s  prospects  for  making  its  next  scheduled  flight? 

♦  Key  points  in  making  the  impoundment  arrangements: 

■  Assign  an  impound  official 

■  Assign  and  assemble  the  flight  control  impound  team  for  this  plane/problem 

■  Assign  or  designate  the  Team  Chief 

■  Coordination  with  QA/QC  (with  additional  documentation) 

■  Release  the  aircraft  for  maintenance  work 

4.1.7  Maintenance  Setup/Preparation 

♦  Where  will  we  park  the  aircraft  to  work  this  reported  problem? 

♦  How  do  we  get  the  aircraft  into  position  (i.e.,  parking  spot)  to  begin  work? 

♦  What  aerospace  ground  equipment  (AGE)  is  on  hand/accessible  for  this  effort? 

♦  Does  additional  AGE  have  to  be  obtained  (e.g.,  from  other  units)  before  we  can  proceed? 

♦  What  tools  and  test  equipment  will  be  needed? 

♦  What  tech  data  (job  guides,  wiring  diagrams,  or  other  reference  aids)  will  be  needed? 

♦  Do  we  need  to  obtain  any  of  the  required  resources  (tools,  test  equipment,  etc)  from  outside  our  unit? 
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♦  Do  the  Aero  Repair  (AR)  riggers  need  to  inspect  the  aircraft  at  this  stage?6 

4. 1.7.1  Troubleshooting  Cycle 

♦  Does  the  problem  identification  give  us  a  head  start  on  diagnosis? 

♦  When  can  the  aircraft  return  to  duty? 

♦  What  detailed  in-flight  data  can  be  accessed  to  shed  light  on  the  reported  fault? 

♦  Are  there  any  pilot-triggered  data  “snapshots”  available  for  inspection? 

♦  Is  any  such  data  volatile  (and  hence  needing  to  be  captured  immediately)? 

♦  What  clues  (informative  symptoms,  etc.)  can  we  glean  at  the  outset? 

♦  Does  the  aircraft  offer  built-in  test  (BIT)  capabilities? 

♦  If  so,  what  formal  maintenance  codes  (e.g.,  BIT  codes)  do  we  get  when  testing  the  aircraft? 

♦  If  not,  what  test(s)  should  we  run  to  obtain  data  on  the  reported  fault  condition? 

♦  How  do  we  translate  BIT/test  results  so  that  they  map  onto  the  structured  diagnostic  aids  (e.g.,  fault 
trees)? 

♦  What  options/recommendations  are  obtained  when  tracing  the  decision  aid  using  the  input  data? 

♦  In  the  case  of  broadly  defined  data  (e.g.,  BIT  codes  indicative  of  a  range  of  possible  conditions),  how 
do  we  sort  out  the  most  likely  culprit? 

♦  After  iterating  through  the  decision  aid’s  inference  structure,  what  is  the  apparent  diagnosis? 

♦  Is  this  diagnosis  consistent  with  what  we  know  about  the  reported  fault  and  this  aircraft? 

♦  If  so,  what  repair  or  replacement  actions  are  needed? 

♦  If  not,  what  should  we  do  to  backtrack  to  either  explore  other  unexplored  paths  in  the  diagnostic  aid’s 
inference  network,  or  else  conduct  tests  to  gather  additionai/better  data  on  die  flight  control  systems? 

♦  Do  we  finally  arrive  at  a  positive  diagnosis  (perhaps  after  multiple  attempts)? 

♦  If  not,  fall  back  to  “swapology”  (swapping  out  components  to  see  if  a  replacement  fixes  the  fault).7 

♦  Do  we  need  to  call  in  the  AR  people? 

4.1. 7. 2  Solution  Reporting/Documentation 

♦  What  do  we  have  to  do  to  document  what  we  did? 

♦  Identify  and  document  what  action  or  change  fixed  the  reported  problem 


6  AR  might  be  involved  on  a  required  or  optional  basis  throughout  the  flight  controls  maintenance  process  path.  For  example,  F-l  5  SMEs  at 
Nellis  noted  the  AR  (rigging)  team  typically  runs  checks  during  or  in  parallel  with  the  maintenance  setup  and  preparation.  At  this  step  in  the 
process,  these  checks  can  induce  a  delay.  If  the  Initial  AR  checks  are  inconclusive  or  conflicting,  the  maintenance  team  may  have  to  wait  for  a 
second  AR  team  to  repeat  die  rigging  checks  and  render  a  second  opinion. 

7  The  term  “swapology”  denotes  an  approach  to  repair  via  changing  out  components  until  tilings  are  OK  again.  This  notion  of  repair-via- 
replacement  came  up  repeatedly  during  the  KA  visits,  and  “swapology”  ended  up  being  a  valuable  piece  of  terminology. 
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♦  Identify  and  document  any  and  all  components  we  changed  out  during  this  maintenance  process 

♦  Assure  that  follow-on  checks  are  or  will  be  done 

♦  Take  care  of  information  system  documentation  requirements  (i.e.,  CAMS) 

♦  Document  this  maintenance  action  in  the  logbook 

♦  Complete  the  various  forms  associated  with  this  aircraft 

♦  Settle  up  with  supply  and  complete  any  associated  documentation  of  supply/requisition  actions  taken 

4.1. 7. 3  Solution  Validation/Verification  &  Completion  Decision 

♦  When  can  we  schedule  a  meeting  to  determine  and  sign  off  on  completion? 

♦  Who  needs  to  be  in  attendance?8 

♦  What  is  the  set  of  documentation  and  forms  we  need  to  process  via  this  meeting? 

♦  What  was  the  course  and  status  of  the  given  maintenance  work  (as  presented  by  the  meeting 
participants  and  the  relevant  documentation)? 

♦  Does  the  Maintenance  Group  Commander  sign  off  on  the  repairs  as  presented? 

4.1. 7.4  Maintenance  Stand-Down 

The  SMEs  did  not  cite  any  key  informational  questions  or  issues  to  be  addressed  in  standing  down  the 
maintenance  process  (e.g.,  cleaning  up,  putting  things  away,  etc.). 

4.1. 7. 5  Unit  Coordination  (Getting  the  Aircraft  Back  to  Duty) 

♦  What  needs  to  be  done  to  prepare  the  aircraft  for  its  next  mission  (e.g.,  doing  mission-specific 
reconfiguration)? 

♦  What  arrangements  need  to  be  made  (e.g.,  towing)  for  returning  the  aircraft  to  duty  status? 

The  implications  of  the  key  questions  and  issues  solicited  from  across  the  SME  groups  will  be  discussed 
further  in  the  subsequent  sections  on  cognitive  task  analysis  and  information  requirements. 

4.1.8  Issue:  Truncating  or  Avoiding  the  Process  Path  in  “No-Fix” 
Situations 

The  process  path  illustrated  in  Figure  1  represents  the  entire  end-to-end  progression  for  a  typical 
maintenance  cycle.  The  completion  and  documentation  of  such  a  maintenance  cycle  is  what  gets  captured 


*  This  meeting  always  includes  the  Maintenance  Group  Commander  (or  designee),  the  impound  official,  and  the  impound  Team  Chief.  Additional 
people  (e.g.,  Production  Superintendent,  Quality  Assurance  representative)  may  participate  depending  on  die  circumstances. 
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in  unit  and  aggregate  performance  statistics.9  However,  it  would  appear  that  such  statistics  underreport  the 
level  of  effort  expended  by  maintenance  staff  in  servicing  their  aircraft.  One  of  the  most  significant 
surprises  encountered  in  the  KA  work  was  the  fact  that  a  substantial  proportion  of  apparent  flight  control 
faults  end  up  being  resolved  without  going  through  this  process  path.  Perhaps  just  as  important  is  the  fact 
that  such  incidents  typically  go  unreported.  This  means  maintainer  time  and  effort  is  expended  on 
diagnosis  and  problem  solving  that  is  not  reflected  in  the  maintenance  statistics.  This  issue  arose  in  the 
very  first  KA  session,  and  was  immediately  added  to  the  list  of  points  to  probe  in  subsequent  sessions. 

Among  the  SME  groups,  the  following  situations  were  cited  as  resulting  in  “no-fix”  resolutions: 

♦  The  apparent  flight  control  problem  is  simply  a  matter  of  “switchology  ”  (incorrect  switch,  knob,  or 
control  settings).  Currently-apparent  problems  can  be  resolved  by  merely  reconfiguring  switch 
settings,  while  past  such  problems  can  be  reasonably  explained  away  by  reference  to  incorrect 
settings.  Such  switchology  faults  were  the  category  of  “no-fix”  situations  most  commonly  cited  by  all 
the  SME  groups. 

♦  The  flight  control  problem  indications  disappear  upon  conducting  a  reset  of  the  flight  control 
(subsystem  or  an  associated  (subsystem.  Particularly  for  those  aircraft  whose  flight  control  systems 
are  computer  controlled,  the  appearance  of  a  flight  control  fault  may  reflect  a  computer  anomaly  and 
not  a  problem  with  the  controls  per  se.  The  SMEs  indicated  that  conducting  a  reset  action  and  then 
rechecking  the  system(s)  often  eliminates  the  apparent  fault. 

♦  The  reported  flight  control  problem  cannot  be  observed  during  reasonable  attempts  to  replicate  it. 
This  is  a  particular  problem  with  flight  control  anomalies,  because  they  may  only  be  apparent  in  the 
air  and  their  circumstances  of  occurrence  may  not  be  capable  of  replication  or  simulation  on  the 
ground. 

♦  The  reported  flight  control  problem  indications  (e.g,  anomalous  in-flight  behavior)  can  be  readily 
explained  by  pilot  actions  inappropriate  or  out-of-range  with  respect  to  established  standards  or 
limits.  This  situation  was  cited  by  multiple  Nellis  SME  groups.  They  explained  that  the  training  and 
testing  missions  conducted  at  that  base  often  put  pilots  into  unfamiliar  situations,  and  that  erroneous 
pilot  actions  often  accounted  for  apparent  flight  control  faults. 

A  summary  of  the  specific  SME  estimates  of  such  “no-fix”  occurrences  is  compiled  in  Table  5. 


9  The  operational  fighter  statistics  summarized  in  Appendix  L 
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Table  5:  SME  Estimates  for  "No  Fix"  Occurrences 


Aircraft10 

Estimates/Comments  Made 

Consensus 

A-10 

“Switchology” 

♦  “No-fix”  incidents  are  at  least  as  frequent  as  incidents 
requiring  actual  maintenance  work. 

♦  Switchology  accounts  for  the  majority  of  “no-fix”  flight 
control  incidents 

50%  (of  all  flight 
control  problem 
reports  turning  out  to 
be  “no-fix”) 

F-16 

“Switchology” 

♦  50  -  60%  of  all  initial  flight  control  problem  reports 

♦  60%  of  all  initial  flight  control  problem  reports 

♦  25  -  50%  of  all  initial  flight  control  problem  reports 

50  -  60% 

F-15 

The  F-15  SMEs  acknowledged  significant  proportional 
occurrence  of  “no-fix"  cases  as  well  as  one  root  cause  being 
“switchology”;  however,  they  didn’t  estimate  incidence  in  %- 
age  terms. 

No  Quantitative 
Estimate  Obtained11 

C-17 

“Switchology” 

♦  10%  of  all  reported  flight  control  problems 

♦  20%  of  all  reported  flight  control  problems 

No  Consensus 
Achieved 

C-17 

“Resets”12 

♦  25%  or  more  of  flight  control  “freezes”  can  be  resolved 
by  resetting  communication  control  units  (CCUs). 

♦  Most  mission  computer  (MC)  problems  can  be  resolved 
with  a  reset. 

♦  Majority  of  problems  with  electronic  flight  instrument 
system  (EFIS)  can  be  resolved  with  reset. 

No  Consensus 

Achieved 

As  Table  5  illustrates,  there  are  differences  among  the  SME  groups  with  respect  to  the  types  of  “no-fix” 
conditions  emphasized  and  the  relative  incidence  they  were  willing  to  attribute  to  them.  However,  the  fact 
remains  that  all  SME  groups  indicated  a  significant  proportion  of  initially  reported  flight  control  problems 
are  resolved  without  having  to  enter  the  plane  into  the  formal  maintenance  cycle  as  laid  out  in  Figure  1. 
With  cursoiy  estimated  “no-fix  occurrence  rates”  ranging  from  a  low  of  10%  (C-17)  to  a  high  of  50%  (A- 
10;  F-16),  this  further  implies  that  a  significant  proportion  of  maintainer  problem  resolutions  are  not 
being  documented.13 


10  No  such  estimates  were  obtained  for  the  RQ-1  and  the  F/A-22.  The  former  has  no  onboard  cockpit,  and  the  latter  is  only  now  arriving. 

11  The  inability  to  obtain  estimates  from  the  F-15  SMEs  may  be  related  to  the  fact  that  the  main  F-1S  session  ended  up  involving  a  large  number 
of  participants  and  became  relatively  unwieldy.  However,  they  were  advised  of  the  striking  estimates  of  approximately  50%  “no-fix  rates”  given 
by  the  A-10  and  F-16  SMEs,  and  they  did  not  discount  or  refute  these  estimates. 

12  Owing  to  die  interconnectedness  of  multiple  electronic  systems  on  their  aircraft,  the  C-17  SMEs  indicated  the  majority  of  “no-fix”  flight 
control  cases  involved  resetting  one  or  more  of  these  associated  systems  (as  opposed  to  die  basic  cockpit  “switchology”  emphasized  by  the  other 
©roups). 

3  Some  SMEs  also  noted  that  “quick-fix”  actions  in  response  to  “red  balls”  commonly  go  undocumented,  especially  when  done  at  the  last  minute 
(right  before  takeoff).  It  was  not  clear  whether  such  “quick  fixes”  are  entirely  subsumed  under  the  “no-fix”  categories  listed  above. 
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4.1.9  Issues  and  Concerns  Regarding  Current  Flight  Control 
Maintenance  Information  Resources 

Having  reviewed  the  flight  control  maintenance  process  path  and  the  key  questions/issues  associated  with 
it,  attention  would  turn  to  the  manner  in  which  current  information  resources  effectively  supported  the 
maintainers.  In  the  following  subsections  an  overview  of  the  issues  cited  with  regard  to  various  data  and 
information  resources  is  provided. 

4.1. 9.1  Operational  Reference  Aids 

This  category  refers  to  job  guides,  checklists,  TOs,  and  other  general  reference  information  employed  in 
the  operation  of  a  given  aircraft.  The  SMEs  consistently  made  reference  to  problems  with  these  reference 
materials,  particularly  in  the  context  of  differences  between  pilot  checklists  and  maintenance  TOs. 
Specific  points  illustrated  this  class  of  issues  included: 

♦  The  general  availability  and  perceived  quality  of  the  primary  reference  documentation  (the  TOs) 
varies  from  aircraft  to  aircraft,  with  the  newest  aircraft  being  the  ones  with  the  least  effective  TO 
support.14 

♦  The  pilots  and  the  maintainers  are  typically  operating  with  different  information  resources.  Pilots  do 
not  use  the  TOs.  Conversely,  pilot  checklists  are  not  used  or  accessed  by  the  maintainers.  Maintainers 
do  not  routinely  see  pilot  checklists  until  and  unless  there  is  an  issue  on  which  they  compare  their 
notes/references.  There  have  been  specific  disjunctions  observed  between  the  maintainers’  and  the 
pilots’  reference  materials.  Most  often  these  consist  of  something  noted  in  one  not  being  mentioned  in 
the  other.  At  the  extreme,  some  of  the  SMEs  specifically  cited  instances  where  both  references 
addressed  the  same  thing,  but  gave  different  data  (e.g.,  values).13 

♦  There  were  repeated  allusions  to  gaps  or  breakdowns  in  the  established  administrative  procedures  for 
documenting  and  correcting  deficiencies  in  basic  reference  materials.  For  all  the  aircraft  addressed, 
there  is  an  administrative  process  through  which  such  corrections  are  made  (via  technical  support 
units).  Problems  with  these  established  processes  include: 

♦  Deficiencies  and  discrepancies  are  not  always  documented  or  not  always  uniformly  documented  and 
submitted  into  the  official  resolution  process. 

♦  Once  reported  to  the  depot  engineering  staff,  maintainers  have  no  visibility  on  the  course  of  efforts  to 
resolve  such  discrepancies/deficiencies. 


14  The  RQ-1  and  F/A-22  SMEs  at  Nellis  both  claimed  their  TOs  were  rudimentary  at  best  and  fragmentary  in  both  coverage  and  depth. 
ls  To  give  one  example,  the  A-10  SMEs  at  Nellis  specifically  cited  a  table  of  data  appearing  in  both  the  maintainers’  TOs  and  die  pilots’ 
references  within  which  different  specific  values  were  listed  for  the  same  parameters). 
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♦  Multiple  SME  groups  stated  the  biggest  problem  with  the  current  TOs  is  not  so  much  the  gaps  and 
deficiencies  identified  with  them,  but  rather  inefficiencies  in  the  mechanisms  for  updating/correcting 
them.16 

♦  Multiple  SME  groups  (and  the  maintenance  Group  Commander  at  Nellis)  indicated  that  much  of  the 
usable  knowledge  on  flight  control  maintenance  is  not  available  within  the  TOs.  Such  knowledge  is 
typically  the  sort  of  “lore”  that  derives  from  experience.  This  “lore”  often  includes  information 
obtained  at  substantial  cost  (in  time  and  effort),  which  will  only  cost  others  as  much  to  find  out  for 
themselves. 

♦  There  is  a  notable  lack  of  channels  or  means  by  which  maintainers  can  share  such  experiential 
knowledge  or  “lore.”  Within  a  given  maintenance  unit,  the  logbook  provides  the  main  repository  and 
channel  for  sharing  such  knowledge.  Across  units  there  are  few  if  any  ways  to  capture  and  share  such 
experiential  knowledge. 

♦  It  is  occasionally  the  case  that  the  TOs  fail  to  provide  adequate  descriptive  or  explanatory  information 
on  a  key  aircraft  component. 

♦  It  is  occasionally  the  case  that  the  TOs  fail  to  provide  adequate  reference  support  for  diagnostic 
procedures  for  one  or  another  subsystem.17 

♦  As  time  goes  on,  some  TOs  actually  become  less  detailed  than  earlier  ones,  in  the  sense  that  they  now 
portray  components  whose  internals  were  previously  detailed  (e.g.,  via  schematic  diagrams)  as  mere 
“black  boxes.”  This  means  that  signal  tracing  (and  hence  troubleshooting)  is  more  difficult  and  less 
informative  using  the  current  (versus  the  earlier)  TOs. 

♦  There  are  instances  in  which  the  relevant  TO  is  simply  inaccurate  owing  to  its  data  being  obsolete. 

♦  In  the  absence  of  any  ability  to  do  solid  troubleshooting,  maintainers  (especially  less  experienced 
ones)  have  no  recourse  except  to  resort  to  “swapology”  (changing  out  components  in  hopes  of 
correcting  the  fault). 

4.1. 9.2  Diagnostic  Aids  (General)18 

The  “Maintenance  Mentor”  concept  is  usually  invoked  in  relation  to  reference  aids  for  the  troubleshooting 

and  diagnostic  functions  in  the  maintenance  process  path.  In  accordance  with  this  topical  focus,  particular 


16  Multiple  SMEs  across  the  various  groups  told  stories  of  it  taking  forever  for  clear-cut  updates  or  corrections  to  be  reflected  in  the  formal  TOs. 
In  some  cases,  the  SMEs  made  reference  to  never  having  seen  any  indication  dial  obvious  revisions  submitted  by  diem  personally  had  ever  been 
acknowledged,  much  less  incorporated.  One  SME  cited  a  particular  clear-cut  correction  he  submitted  some  two  years  earlier,  but  which  was  still 
unacknowledged  and  unimplemcnted.  This  lack  of  perceived  action  or  interest  tends  to  de-motivote  maintainers  to  go  to  die  trouble  of  trying  to 
improve  die  TOs. 

17  For  example,  the  F-16  SMEs  stated  their  TOs  offer  poor  support  in  diagnosing  problems  with  the  ALR-56M  electronic  countermeasures 
subsystem(s). 

11  This  section  focuses  on  general  issues  relating  to  diagnostic  aids.  Electronic  (e.g.,  computer-based)  diagnostic  aids  will  be  more  specifically 
discussed  in  a  subsequent  section. 
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attention  was  paid  to  the  availability,  utility,  and  sufficiency  of  diagnostic  decision  aids.  Although  much 

time  was  spent  probing  for  data  on  deficiencies  in  the  diagnostic  aids  employed  in  flight  control 

maintenance,  the  SMEs  offered  relatively  few  complaints  concerning  the  diagnostic  aids  themselves. 

Some  of  the  most  significant  points  made  by  the  SMEs  on  general  diagnostic  aiding  issues  included: 

♦  There  are  a  variety  of  diagnostic  aids  available  to  maintainers,  including  the  TOs  themselves,  fault 
isolation  manuals  (FIs),  job  guides,  schematic  diagrams,  CAMS  history,  precedents  in  the  unit 
logbook,  BIT  cards,  and  fault  trees. 

♦  Even  for  the  more  common  paper  aids,  there  are  variations  in  the  types  of  reference  documentation 
available  from  one  aircraft  to  another. 

♦  Depending  on  the  aircraft  type,  the  maintenance  unit,  the  maintenance  environment  and  other 
circumstances,  the  precise  mix  of  available  paper  and/or  electronic  diagnostic  aids  will  vary.19 

♦  For  those  aircraft  where  electronic  aids  are  available,  it  can  still  be  the  case  that  these  aids  cannot 
actually  be  deployed  for  use.20 

♦  Electronic  aids  require  laptops  (or  similar  devices)  on  which  to  run  the  aiding  applications. 
Application  of  electronic  aids  therefore  remains  dependent  on  the  availability  and  serviceability  of 
these  platforms  from  day  to  day 21 

♦  The  fault  isolation  guides  serve  as  the  primary  “logic”  being  followed  in  diagnosing  a  problem.  The 
maintainers  match  observed  data  to  conditions  in  the  FI  structure,  then  proceed  by  following  where  in 
that  logical  layout  the  identified  state  or  condition  leads. 

♦  The  availability  of  built-in  test  (BIT)  code  data  greatly  expedites  the  diagnostic  process.  For  one 
thing,  BIT  codes  are  usually  more  quickly  obtained  than  (e.g.)  visual  or  manual  inspection  data.  For 
another  thing,  the  BIT  codes  are  mappable  into  a  finite  set  of  formal  categories  and  alternatives. 

♦  Both  paper  FIs  and  available  electronic  diagnostic  aids  have  been  found  to  contain  deficiencies,  gaps, 
and  even  some  errors. 

♦  The  procedural  cycle  for  reporting,  revising,  and  reissuing  FI  aids  is  similar  to  that  established  for  the 
TOs  (as  discussed  earlier). 

♦  The  lack  of  feedback  and  slowness  of  this  revision  cycle  noted  earlier  with  respect  to  TOs  applies  to 
the  FI  materials  as  well. 


“  The  maintenance  aid  toolkit  for  the  newest  manned  aircraft  surveyed  (the  C-17  and  F/A-22)  displays  a  marked  emphasis  on  electronic  aid 
deployment  The  F/A-22  is  the  most  radical  example  in  this  regard,  because  its  maintenance  concept  revolves  around  the  computerized  Portable 
Maintenance  Aid  (PMA)  as  its  sole  intended  maintenance  aid. 

20  For  example,  the  C-17  maintainers  identified  die  Digital  Technical  Order  System  (DTOS)  as  their  most  valuable  documentation  aid.  However, 
DTOS  is  deployed  on  a  digital  disc  (DVD).  The  onboard  C-17  portable  computers  do  not  have  DVD-capable  drives,  thus  making  it  impossible  to 
simply  carry  a  DTOS  disc  onto  the  aircraft  for  immediate  use.  As  a  result,  the  C-17  maintainers  have  to  bring  another  laptop  onto  the  plane. 

For  example,  die  F-15  SMEs  at  Nellis  stated  that  laptops  are  not  guaranteed  to  be  available  if  too  many  other  parties  are  using  them.  In  other 
words,  availability  of  a  laptop  through  the  support  section  can  make  a  difference  in  whether  or  not  a  maintenance  task  can  proceed. 
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♦  Those  maintainer  groups  with  equivalent  access  to  both  paper  and  electronic  fault  diagnosis  aids 
indicated  a  general  preference  for  the  paper  materials  in  everyday  use.22 

♦  Strictly  following  the  FIs  can  lead  to  states  where  the  maintainers  are  instructed  to  stop  and  “check 
rig”  (i.e.,  call  in  the  riggers).  Such  stopping  places  are  time-consuming,  because  the  riggers  are  an 
independent  team,  which  is  not  usually  included  in  the  impoundment  team. 

♦  Tips  and  clues  derived  from  experience  can  enhance  the  utility  of  the  FIs.23 

4.1. 9.3  Diagnostic  Aids  (Electronic) 

This  section  will  outline  the  major  points  the  SME  interviewees  cited  with  respect  to  electronic  aiding. 

Some  of  the  most  significant  points  made  on  electronic  diagnostic  aiding  issues  included: 

♦  The  utility  of  electronic  diagnostic  aids  is  largely  dependent  on  the  diagnostic  software  they  contain. 

♦  Issues  surrounding  the  software  itself  can  constrain  the  basic  availability  of  the  aid  for  maintenance 
use,  particularly  during  the  period  when  software  is  first  made  available.24 

♦  Some  maintainers  are  generally  “computer  averse,”  unfamiliar  with  computer  usage,  or  less  adept  at 
using  electronic  versus  paper  aids. 

♦  The  legibility  of  laptop-based  aids  varies  with  circumstances  and  is  often  a  cause  of  diminished 
utility.  The  most  commonly  cited  such  problem  relates  to  severe  glare  off  the  display  screen  under 
certain  lighting  conditions,  especially  direct  sunlight. 

♦  Another  problem  with  electronic  aid  displays  was  small  fonts  that  make  it  difficult  to  read  text 
information  or  diagram  captions.  Reading  such  small  fonts  requires  the  user  to  get  very  close  to  the 
display  (further  increasing  the  probability  he/she  cannot  use  the  aid  while  working  directly  on  the 
aircraft).25 

♦  Maintenance  work  is  often  two-handed  work.  This  means  that  maintainers  must  set  a  laptop  aid 
somewhere  and  then  go  back  and  forth  between  their  manual  actions  on  the  aircraft  and  their  manual 
interactions  with  the  laptop.  At  the  very  least,  there  is  a  lot  of  head  turning  required  to  address  both 
the  aircraft  and  the  laptop.  This  divides  attention  and  consumes  time. 

♦  Simply  positioning  the  laptop  can  be  a  problem.  Often  a  convenient  surface  cannot  be  found  close  to 
the  relevant  area  on  the  aircraft  where  die  laptop  can  be  conveniently  and  safely  placed. 


“The  most  commonly  cited  bases  for  this  preference  related  to  the  feet  that  paper  aids  are  more  portable,  less  fragile,  less  demanding  of 
digressive  actions  to  use  (a  glance  versus  typing  and  mouse-clicking),  and  free  from  the  usability  problems  associated  with  the  laptop-based  aids. 

For  example,  the  Nellis  Aero  Repair  SMEs  claimed  the  F-15E  FIs  are  somewhat  better  than  those  for  the  F-15C  because  they  provide  more 
detailed  information,  especially  more  tips  and  data  derived  from  past  experience. 

24  For  example,  F-15  SMEs  at  Nellis  noted  that  newly  arrived  software  tools  are  still  unavailable  for  flightline  use  pending  resolution  of  issues 
relating  to  validation  and  security. 

25  This  text  legibility  issue  was  noted  by  multiple  SME  groups.  It  seemed  to  be  a  common  issue  with  die  C-17  maintainers  and  the  EDNA  (F-16) 
users. 
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♦  None  of  the  aircraft  surveyed  originally  made  provisions  for  laptop  placement/positioning.26 

♦  Some  workarounds  or  adaptations  have  been  made  for  laptop  placement.27 

♦  The  electronic  aids  are  often  touted  as  being  “portable,”  but  they  are  often  less  portable  (or  less 
convenient)  than  paper  materials.28 

♦  The  procedural  and  interface  structure  of  some  of  the  laptop-based  diagnostic  aids  is  such  that  it  can 
be  more  cumbersome  to  navigate  through  the  fault  tree  on  the  laptop  than  using  a  reference  card.  This 
can  make  it  more  time-consuming  to  use  the  electronic  aid  than  a  paper  equivalent.29 

♦  The  software  used  for  electronic  diagnostic  aids  is  often  tied  to  one  or  another  operating  system.  This 
linkage  results  in  constraints  regarding  what  platform  must  be  available  to  use  the  software  or  even 
whether  the  software  can  be  used  at  all. 

♦  Version  discrepancies  among  software-based  reference  and  diagnostic  aids  are  a  persistent  headache. 
It  is  often  the  case  that  different  personnel  or  units  are  operating  with  different  versions,  and  hence 
with  different  specific  data. 

4.1. 9.4  Maintenance  Documentation  Aids 

In  addition  to  their  diagnostic  assets,  maintainers  must  have  some  means  for  documenting  actions  during 

the  course  of  maintenance  (e.g.,  requirements,  procedural  notes,  etc.).  Some  of  the  points  made  by  the 

SMEs  on  documentation  aiding  issues  included: 

♦  Most  documentation  is  still  done  by  hand  on  paper  (e.g.,  AFTO  78 1 A  entries,  logbook  entries;  notes 
on  clipboards). 

♦  Most  documentation  is  generated  in  the  course  of  the  maintenance  process  in  the  actual  maintenance 
setting. 

♦  As  a  result,  using  a  computer  for  these  documentation  functions  is  (or  would  be)  subject  to  the  same 
usability  problems  noted  earlier  for  computer-based  diagnostic  aids. 

♦  When  asked  if  they  saw  any  potential  for  improved  support  by  computerizing  incidental 
documentation,  the  SMEs  consistently  answered  in  the  negative.30  The  one  significant  exception  to 
this  trend  was  die  F/A-22  group. 31 


26  This  is  particularly  striking  on  the  F/A-22,  for  which  the  PMA  is  the  maintainers’  sole  means  for  interacting  with  the  aircraft  The  PMA 
connection  points  provided  (in  the  cockpit  and  each  of  the  main  wheel  wells)  offer  no  support  for  the  PMA  unit. 

27  The  C-17  SMEs  at  Charleston  noted  that  repeated  complaints  about  laptop  placement  inside  the  aircraft  had  led  to  the  installation  of  a  shelf  for 
this  purpose.  F/A-22  maintainers  at  Edwards  AFB,  CA  custom-built  a  pedestal  stand  on  which  to  place  the  PMA.  This  innovation  is  being 
evaluated  for  possible  proliferation. 

22  There  is  no  electronic  aid  as  light  (and  hence  portable)  as  the  FI  quick  reference  cards.  F-I6  SMEs  at  Nellis  noted  that  it  is  just  as  easy  to  lug  an 
entire  manual  out  to  the  plane  as  a  laptop.  The  problem  is  particularly  evident  with  the  F/A-22  PMA,  a  portable  computer  unit  whose  hardware  is 
basically  over  a  decade  old.  About  the  size  of  an  old-fashioned  reel-to-reel  tape  recorder,  die  PMA  is  an  example  of  a  device  that  is  “luggable” 
rather  than  “portable”. 

29  In  one  F-16  session  at  Nellis,  an  SME  bluntly  claimed  he  could  navigate  through  the  paper  manual  5  times  faster  than  the  equivalent  material 
on  the  computer-based  aid.  The  others  in  the  session  agreed. 
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4.1. 9.5  Histories I  Dete  on  Msintensnee  Processes  end  Procedures 

Multiple  SMEs  cited  the  usefulness  of  knowing  how  a  particular  maintenance  problem,  especially  a  very 
challenging  one,  was  resolved  in  the  past.  In  the  case  where  such  knowledge  is  extrinsic  to  the  formal 
reference  aids  (e.g.,  the  TOs  and  FIs),  it  is  part  of  the  experiential  knowledge  discussed  later.  However, 
such  historical  information  can  be  similarly  useful  in  the  formal  diagnostic  procedures  performed  within 
the  purview  of  the  established  TOs  and  FIs.  For  example,  the  F- 16  “Falcon”  unit  SMEs  at  Nellis  noted  the 
growing  importance  they  attribute  to  documenting  the  course  of  die  troubleshooting  activities  themselves 
(as  opposed  to  only  the  starting  states  and  the  outcomes).  They  claimed  the  particular  inference  chain 
followed  in  the  TO/fault  tree  can  itself  prove  informative  as  a  future  point  of  reference  in  subsequent 
similar  situations.  They  stated  that  they  even  sometimes  record  what  was  connected/disconnected  during 
the  troubleshooting  process  as  part  of  this  historical  trace. 

4.1.10  Issues  and  Concerns  Regarding  Tools  and  Instruments 
Employed  in  Flight  Control  Maintenance 

As  intended  in  the  KA  plan,  the  SME  groups  were  probed  for  issues  and  concerns  relating  to  their  tools 
and  instruments.  These  include  devices  employed  in  capturing,  testing,  and  manipulating  aircraft 
parameters.  These  actions  yield  data  that  is  then  fed  into  or  compared  against  the  diagnostic  logic 
available  through  the  maintainers’  personal  knowledge  and  their  diagnostic  aids.  Such  tools  and 
instruments  may  be  peripheral  to  the  cognitive  or  decision  processes  at  the  core  of  diagnosis,  but  they  are 
critical  to  accomplishment  of  those  processes.  One  cannot  decide  which  next  step  to  talm  in  a  fault  tree 
until  and  unless  the  condition(s)  specified  in  the  diagnostic  aid  are  verified  (by  test  and  data  analysis). 

This  verification  cannot  be  done  until  and  unless  the  equipment  allows  one  to  conduct  whatever  testing  is 
required  to  obtain  or  check  the  relevant  data. 

It  was  not  surprising  that  the  SME  groups  consistently  cited  problems  relating  to  such  testing  equipment. 
However,  it  was  surprising  that  the  frequency  of  such  citations  and  the  importance  attached  to  them 
surpassed  the  issues  and  problems  reported  with  the  diagnostic  guides  themselves.  Some  of  the  more 
significant  such  points  included  the  following: 


30  One  F-16  group  noted  this  had  in  effect  been  tried  before.  They  stated  CAMS  had  been  made  available  on  the  flightline  so  maintainers  could 
enter  maintenance  activity  data  as  they  went  along.  This  SME  group’s  characterization  of  that  experiment  was  “a  disaster.”  They  <*«M  the  cited 
usability  issues  were  the  reason  for  failure  of  that  initiative. 

31  However,  there  are  some  relevant  factors  that  must  be  borne  in  mind  in  considering  their  situation.  First,  their  PMA  support  tool  is  specifically 
designed  to  serve  as  an  electronic  repository  for  notes  mid  comments  generating  during  its  use.  This  means  they  have  a  capability  for  dynamic 
electronic  documentation  not  available  to  the  other  maintainers  interviewed.  Second,  the  PMA  deployment  concept  mandates  that  they  use  this 
device  exclusively.  As  such,  they  do  not  have  a  choice. 
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♦  Some  designated  test  equipment  is  not  very  useful,  but  the  maintainers  have  to  use  it  anyway.32 

♦  Maintainers  do  not  have  much  confidence  in  some  of  the  test  equipment.33 

♦  Much  of  the  test  equipment  is  aged  and/or  dysfunctional.34 

♦  Newer  is  not  necessarily  better  in  the  case  of  some  devices.  Improvements  in  one  aspect  of  a  device’s 
capabilities  (e.g.,  measurement  precision)  are  sometimes  obtained  only  with  additional  problems  with 
another  aspect  (e.g.,  its  usability).35 

♦  For  some  specialized  devices  there  is  only  one  on  hand,  meaning  that  they  can  constitute  “single  point 
of  failure  conditions”  in  being  able  to  conduct  certain  maintenance  tasks.36 

♦  Most  of  the  SME  groups  at  Nellis  complained  of  AGE  being  quite  old  and  subject  to  malfunction. 
Such  comments  were  less  frequently  made  at  Charleston,  where  the  maintainers  enjoy  an  inventory  of 
newer  equipment. 

♦  Some  common  flight  control  problem  components  are  not  subject  to  troubleshooting  because  there  is 
no  equipment  available  for  the  purpose.37 

♦  Some  of  the  available  aircraft  test  equipment  cannot  be  used  because  of  connection  or  other 
compatibility  issues.38 

♦  In  some  cases  there  are  newer,  more  sophisticated,  and/or  better  pieces  of  equipment  commercially 
available.39 

♦  Better  equipment  cited  by  the  SMEs  is  already  in  use  elsewhere,  even  within  USAF  and  other  military 
units.40 


32  For  example,  the  FLTS  tester  is  the  only  available  flight  control  tester  available  for  the  F-15C.  Unfortunately,  the  F-15C  SMEs  indicated  that 
even  when  it  works  properly  it  provides  a  limited  range  of  testable  parameters  and  the  data  obtained  is  often  unreliable. 

33  For  example,  the  AR  (rigger)  SMEs  at  Nellis  stated  they  do  not  trust  the  results  from  a  FLTS  tester.  In  particular,  they  stated  FLTS  testing  can 
“pass”  a  component  multiple  times,  even  though  alternative  tests  clearly  show  the  component  to  be  unserviceable. 

34  For  example,  Nellis  Aero  Repair  has  a  force  tester  for  measuring  G-force  loads  on  flight  control  components;  however,  it  is  old  and  very 
unreliable.  The  F-16  SMEs  reported  their  TTU-205  units  are  often  broken  or  out  of  calibration.  Although  a  sufficient  number  is  available,  few  if 
any  are  usable  when  needed. 

33  A  good  example  of  this  situation  was  cited  by  Nellis  AR  people.  Their  available  stick  rigging  indicator  tools  have  changed  in  recent  years,  but 
not  completely  for  die  better.  The  old  (mechanical)  devices  took  several  minutes  to  install,  whereas  the  newer  (digital)  ones  only  require  about  30 
seconds.  However,  the  new  digital  versions  are  fragile,  have  problems  with  the  batteries  falling  out,  and  can  be  expected  to  last  only  about  4 
months.  The  old  mechanical  versions  last  for  years. 

36  To  give  an  extreme  example,  the  Nellis  Aero  Repair  Shop  is  small,  and  it  is  has  a  minimal  inventory  of  test  equipment  and  took.  In  general, 
they  have  only  one  of  each  tool  on  hand  at  any  given  time.  Moreover,  some  of  these  took,  such  as  a  cable  tensiometer,  are  the  only  ones  on  base. 
This  means  that  if  a  piece  of  test  equipment  breaks  or  is  out  for  recalibration  Aero  Repair  has  no  testing  capability.  This  can  bring  flight  control 
work  to  a  halt 

37  Two  good  examples  were  cited  by  the  F-16  and  F-15  training  SMEs  at  the  Nellis  FTD.  Actuators  were  repeatedly 
cited  as  a  source  of  diagnostic  headaches  on  both  aircraft.  However,  there  is  no  diagnostic  aid  for  troubleshooting 
integrated  servo-actuators  and  no  tester  available  for  checking  them  on  either  plane. 

F-16  leading  edge  flaps  incorporate  numerous  mechanical  subcomponents,  so  troubleshooting  them  quickly  gets 
complicated,  but  there  is  no  tester  available  to  help  with  this  job. 

38  For  example,  in  one  F-16  session  it  was  noted  that  the  only  available  pitot-static  tester  (the  venerable  TTU-205)  cannot  even  be  connected  to 
the  F-16,  because  the  ones  on  hand  do  not  have  any  F-16  compliant/compatible  connectors. 

39  For  example,  an  F-16  AFETS  SME  noted  a  commercially  available  tester,  the  ADTS  405F,  that  is  for  superior  to  the  TT-205  for  aircraft  pitot- 
static  checks. 

40  Some  AF  Reserve  and  Air  National  Guard  units  are  already  using  the  Druck  405  tester. 
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♦  Some  equipment  proven  consistently  useful  is  not  always  officially  available  or  officially 
sanctioned.41 

4.1.11  Connectors  and  Connections:  A  Persistent  Issue  in  Both 
Diagnosis  and  Equipment  Usage 

One  point  that  surfaced  multiple  times  during  the  KA  sessions  was  the  extent  to  which  connectors  and 

connections  (as  contrasted  with  the  components  being  connected)  caused  headaches  for  maintainors  This 

includes  problems  with  connectors  on  the  aircraft  and  connectors  employed  in  setting  up  the  test  devices. 

Illustrative  SME  comments  on  this  subject  included: 

♦  Connectors  are  quite  often  the  source  of  fault  conditions  on  die  aircraft.42 

♦  Many  of  the  problems  associated  with  AGE  involve  the  connections  between  these  devices  (e.g., 
hydraulic  mules)  and  the  aircraft. 

♦  Variations  and  glitches  in  cables  and  connectors  can  affect  test  readings  and  hence  mislead 
troubleshooters  (at  the  cost  of  time  and  effort). 

♦  The  SMEs  and  the  support  sections  repeatedly  cited  the  TT-205  connection  set  (“hose  kit”)  as  the 
most  common  source  of  problems  in  using  the  tester  43 

♦  The  ubiquity  of  connection  faults  in  aircraft  wiring  is  illustrated  by  the  fact  that  one  of  the  only  tools 
found  in  the  support  sections  surveyed  was  a  wire  and  wire  connection  repair  kit. 

♦  The  EDNA  diagnostic  tool  used  on  the  F- 1 6  is  deployed  in  a  kit  containing  the  basic  cables.  In 
addition,  the  support  section  stocks  a  connector  kit  with  approximately  15  to  20  additional  cables  and 
connectors  for  linking  EDNA  to  a  given  aircraft.  The  box  containing  the  additional  cable  set  is  larger 
than  the  EDNA  kit  itself. 

♦  Sophisticated  onboard  test  capabilities  are  geared  to  identifying  faults  in  discrete  components,  and 
they  cannot  discern  when  faults  lie  in  the  lines  or  connections  among  these  components. 

♦  The  lack  of  compatible  connectors  is  a  major  reason  why  the  TT-205  is  not  used  on  the  F-l  6. 


For  example,  the  F-16  SMEs  cited  breakout  boxes  as  greatly  facilitating  their  ability  to  track  down  signal  faults  in  aircraft  wiring.  They  are 
particularly  helpful  in  determining  which  particular  pin(s)  relate  to  a  problem  identified  in  a  connection  or  connector.  The  use  of  breakout  boxes 
is  not  emphasized,  there  are  no  standard  breakout  boxes  issued,  and  maintenance  units  often  end  up  custom-building  their  own. 

42  The  F-16  SMEs  claimed  the  highest  failure  rates  for  F-16  swappable  components  are  found  among  the  weight  on  wheels  (WOW)  switch 
cannon  plugs. 

43  The  TT-205  hose  kits  are  harder  to  get  replaced  than  the  testers  themselves.  The  most  commonly  reported  failure  had  to  do  with  sealer  rings  in 
the  hose  end  connectors.  If  one  of  these  rings  warps  or  degrades  the  hoses  are  unusable,  making  the  tester  unusable.  Replacement  hose 
components  have  to  be  ordered  from  Canada. 
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4.1.12  Issues  Concerning  Maintenance  Expertise  and  Differences 
Between  Experienced  and  Inexperienced  Maintainers 

The  notion  of  “expertise”  is  important  in  undertaking  task-specific  cognitive  and  information 
requirements  analyses.  With  respect  to  cognitive  factors,  it  is  important  to  identify  what  features  are 
associated  with  attributed  expertise  in  a  given  task.  This  sets  the  stage  for  focusing  on  what  constitutes 
expert  performance  and  aids  in  facilitating  such  performance.  With  respect  to  information  requirements,  it 
is  important  to  both  discern  what  “expert-level”  information  support  is  needed  for  proficient  technicians 
and  to  account  for  more  “novice-friendly”  information  support  to  allow  less  experienced  people  to  operate 
adequately.  During  the  KA  sessions  a  point  was  made  to  ask  the  SMEs  about  what  characterized  expertise 
and  what  differentiated  experts  from  novices. 

4.1.12.1  What  Characterizes  “Expertise”  in  Maintenance  Work? 

The  factors  or  features  cited  as  generally  indicative  of  “expertise”  in  maintenance  work  included: 

♦  Experts  are  more  proficient  at  making  a  “leap”  from  initial  problem  report  to  a  likely  diagnosis. 

♦  Experts  are  quicker  and  more  proficient  at  estimating  the  level  of  effort  for  repairs  and  the  likely 
timeframe  required  to  return  the  aircraft  mission  capable. 

♦  Experts  are  more  proficient  at  making  a  “leap”  from  initial  discrepancy  or  diagnostic  hypothesis  to 
identifying  the  component  that  must  be  replaced. 

♦  Experts  are  more  knowledgeable  about  the  details  of  the  aircraft  (sub)systems  and  how  they 
interoperate. 

4.1.12.2  What  Distinguishes  Expert  from  Novice/Inexperienced 
Maintainers? 

In  the  course  of  the  KA,  it  was  learned  that  distinctions  between  experts  and  novices  not  only  affect  the 
quality  and  course  of  the  maintenance  process,  but  also  affect  die  readiness  with  which  other  parties 
accept  the  results  of  a  maintainer’s  diagnoses.  The  factors  or  distinctions  cited  as  differentiating  expert 
from  novice  maintenance  performance  included: 

♦  Novices  tend  to  scrupulously  follow  TOs  step-by-step. 

♦  Multiple  SME  groups  characterized  novice/younger  maintainers  as  relying  exclusively  on  the  TO  and 
BIT  as  the  entirety  of  their  diagnostic  procedure  (as  opposed  to  digging  into  the  fault  via  exploratory 
troubleshooting). 
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♦  There  is  a  discernible  difference  between  experts  and  novices  in  terras  of  how  they  view  the  TOs. 
Novices  know  only  what  is  in  the  TO  and  are  typically  unable  to  troubleshoot  beyond  what  is  “in  the 
book.” 

■  Experts  are  more  proficient  about  relating  symptoms  to  states  or  features  of  the  physical  aircraft 
layout  (wiring,  data  network  infrastructure,  mechanicals). 

■  Experts  generally  have  more  thorough  knowledge  of  how  the  aircraft  and  its  constituent 
subsystems  actually  work. 

■  Novices  are  notably  less  knowledgeable  and  cognizant  of  details  on  how  the  aircraft  works  (and 
how  its  components  and  subsystems  interact)  as  time  goes  on. 

■  Novices  are  more  reluctant  to  initiate  free  form  exploratory  troubleshooting  when  the  available 
diagnostic  procedures  (e.g.,  tracing  the  feult  tree  via  the  FIs)  fail  to  pinpoint  the  cause  of  a 
problem. 

■  Novices  are  quicker  and  more  amenable  to  stopping  the  maintenance  process  and  “making  the 
call”  (invoking  external  technical  support)  once  the  cookbook  diagnostic  procedure  bogs  down. 

■  The  availability  of  BIT  capabilities  leads  to  both  (a)  increased  reliance  on  the  BIT  tests  and  their 
results  as  well  as  (b)  a  lack  of  proficiency  at  “free  form”  troubleshooting.  These  factors  tend  to 
lead  to  atrophied  or  suboptimal  troubleshooting  expertise. 

■  Because  new  maintainers  are  increasingly  trained  on  aircraft  offering  these  capabilities,  they  are 
never  exposed  to  the  need  for  exploratory  troubleshooting  (beyond  what  the  BIT  codes  and  FIs 
allow). 

♦  More  experienced  maintainers  often  employ  “non-standard”  maintenance  tactics  (e.g.,  leaping  to 
tentative  candidate  solutions;  immediately  doing  “swapology”  as  recommended  from  their 
experience).  Less  experienced  maintainers  are  often  influenced  by  these  non-standard  habits  observed 
in  the  more  experienced  personnel.  Unfortunately,  the  younger  maintainers  emulate  the  expert 
behaviors  without  the  benefit  of  the  deeper  knowledge  upon  which  the  experts  proceed. 

♦  It  is  becoming  more  common  for  maintainers  to  migrate  among  different  aircraft  types  during  the 
course  of  their  maintenance  careers.  This  works  against  gaining  the  experiential  knowledge 
underlying  expert  performance.  For  one  thing,  after  each  migration  the  maintainer  essentially  has  to 
start  from  scratch  in  learning  the  new  aircraft.  As  time  goes  on,  the  maintainers  acquire  less  and  less 
experiential  expertise  with  each  passing  aircraft  on  which  they  work,  thus  diminishing  the  cumulative 
degree  of  general  expertise  carried  with  them  through  subsequent  migrations. 

♦  Lesser-experienced  maintainers,  being  generally  younger,  are  typically  more  comfortable  and  adept  in 
using  computer-based  aids. 
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4.1.13  Maintainers’  #1  Recommendation  for  Information  improvement: 
Better  Situation  Awareness  on  In-flight  Problem  Context 

In  each  session,  SMEs  were  probed  for  their  “wish  lists”  on  what  type(s)  of  data  or  information  they  felt 
would  most  improve  their  ability  to  perform  flight  control  maintenance.  For  example,  SMEs  were 
consistently  asked  what  one  type  of  data  or  information  they  would  desire  to  improve  the  flight  control 
maintenance  process.  As  each  SME  group  gave  their  “wish  list”,  their  general  suggestions  were  noted  and 
were  brought  up  with  subsequent  groups  (in  addition  to  whatever  those  subsequent  groups  cited  as  their 
“wish  list”). 

Throughout  all  the  sessions  at  both  Nellis  and  Charleston,  the  #1  item  on  the  SME  “wish  list”  was  better 
information  on  what  was  happening  in  flight  when  a  flight  control  problem  occurred.  Even  the  riggers 
stated  their  #1  priority  question  is,  “What  did  the  jet  do?”  More  abstractly  phrased,  the  SMEs  stated  the 
single  most  important  improvement  would  be  better  situation  awareness  (SA).  The  availability  and 
completeness  of  such  data  will  vary  with  the  clarity  and  detail  of  the  aircrew’s  fault  reporting.  These 
factors  also  vary  with  the  type  of  aircraft,  because  different  aircraft  have  differing  mechanisms  (if  any)  for 
capturing  in-flight  data  and  the  data  thus  captured  also  varies  from  one  aircraft  type  to  another. 

Additional  discussion  of  this  issue  regarding  cognitive  aspects  of  the  maintenance  process  will  be 
provided  in  the  section  on  CTA.  A  more  detailed  discussion  of  this  issue  with  respect  to  current  and 
prospective  information  assets  will  be  provided  in  the  section  on  information  requirements  analysis. 

4.1.14  Maintainers’  #2  Recommendation  for  Information  Improvement: 
Ability  to  Simulate  In-flight  Conditions  on  the  Ground 

In  the  absence  of  information  detailed  enough  to  immediately  discern  a  fault’s  root  cause,  the  next  best 
thing  would  be  to  simulate  flight  behavior  with  the  aircraft  on  the  ground.  If  such  a  simulation  were 
possible,  it  would  allow  maintainers  to  replicate  and  observe  anomalies  directly,  rather  than  being  forced 
to  rely  on  whatever  information  they  could  obtain  from  the  aircrew  and/or  available  in-flight  data.  By  and 
large,  this  is  difficult  or  impossible  to  do.  More  specific  SME  comments  on  this  topic  included  the 
following: 

♦  The  ability  to  simulate  or  reproduce  a  reported  problem  would  improve  the  diagnostic  process. 

♦  The  ability  to  simulate  in-flight  conditions  on  the  ground  would  help  reduce  “cannot  duplicate” 
discrepancies. 
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♦  The  operation  of  certain  onboard  safety  devices  (most  particularly  the  weight-on-wheels  or  WOW 
switches)  prevents  even  limited  opportunities  to  attempt  simulations  on  the  ground.  On  some  aircraft 
the  WOW  switches  can  be  defeated  to  allow  such  testing,  but  such  practices  are  frowned  upon.44 

♦  Some  relevant  parameters  simply  cannot  be  simulated  on  the  ground  (e.g.,  axes  of  in-flight 
orientation,  G-forces). 

4.1.15  Maintenance  Training:  A  Consistent  Focus  for  Improvement 
Recommendations 

The  subject  of  training  surfaced  again  and  again  during  the  SME  sessions  at  both  Nellis  and  Charleston. 
The  team  had  made  a  point  to  schedule  KA  interviews  with  trainers;  however,  training  was  cited 
repeatedly  in  the  other  (maintainer)  SME  sessions  as  well.  Much  of  the  discussion  on  training 
improvements  mirrored  the  types  of  points  made  with  respect  to  perceived  differences  between  novice 
and  expert  maintainer  capabilities  (see  Section  4.1.12.2  above).  Some  of  the  more  significant  points  made 
in  reference  to  training  issues  included: 

♦  Experts  are  progressively  more  proficient  about  relating  symptoms  to  states  or  features  of  the  physical 
aircraft  layout  (wiring,  data  network  infrastructure,  mechanicals)  than  novices  because  novices,  even 
those  just  out  of  training,  exhibit  a  discemibly  lower  degree  of  knowledge  about  how  the  given' 
aircraft  (or  any  aircraft)  works. 

♦  The  current  flight  control  training  scenarios  typically  involve  simply  stepping  through  the  TO.  In 
other  words,  student  training  is  narrowly  focused  on  cookbook  procedures  and  not  general  diagnostic 
skills  or  experience  in  problem  solving.  This  leads  to  a  situation  in  which  newer/younger  maintainers 
may  know  how  to  do  something,  but  are  at  a  loss  to  understand  why  they  are  doing  it. 

♦  The  Nellis  Aero  Repair  SMEs  noted  a  progressive  deterioration  in  the  ability  of  younger/newer 
maintainers  to  employ  effective  and  accurate  terminology  and  references  to  parameters  in  describing 
aircraft  behaviors  that  must  be  accounted  for.  These  people  recommended  better  training  on  “what  the 
pilots  are  telling  you.” 

♦  Multiple  SMEs  (most  particularly  the  trainers)  generally  characterized  newer  maintainers  as  coming 
out  of  technical  school  with  noticeable  deficiencies  in  general  technical  knowledge  about  how  aircraft 
and  aircraft  subsystems  operate. 


“  For  example,  by  using  a  screwdriver  one  can  disable  the  WOW  switches  on  an  F-16  and  thereby  make  it  possible  to  test  aircraft  systems  as  if 
they  were  airborne.  The  FTD  trainers  who  cited  this  example  clearly  stated  both  (a)  this  tactic  is  extremely  effective  and  (b)  it  is  potentially  very 
dangerous  and  is  only  authorized  under  vexy  specific  circumstances. 
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♦  Other  deficiencies  noted  for  incoming  trainees  included  deficiencies  in  the  ability  to  use  written 
reference  materials,  capacity  for  grasping  abstractions,  and  ability  to  identify  and  project 
ramifications  of  observations  and  data  obtained  during  troubleshooting. 

♦  One  F- 1 5  trainer  stated  that  flight  control  training  attempts  to  address  the  full  range  of  functions 
reflected  in  the  process  path  diagram  in  Figure  1  (e.g.,  the  paperwork  and  CAMS  support  implicit  in 
die  unit  coordination  steps).  Nonetheless,  the  exercises  trainees  perform  typically  focus  on  the  central 
(troubleshooting  cycle)  portion  of  the  process  path. 

♦  In  training,  the  tactic  labeled  “swapology”  (switching  out  parts  to  see  if  that  fixes  the  problem)  is 
characterized  as  bad  practice,  and  students  are  discouraged  from  relying  upon  it. 

♦  There  is  a  significant  problem  inducing  the  motivation  and  imparting  the  expertise  to  dig  into  flight 
control  systems  because  such  detailed  troubleshooting  requires  both  specific  knowledge  on  die  given 
aircraft  and  general  knowledge  of  electrical,  hydraulic,  and  mechanical  technologies. 

♦  Deep  working  knowledge  of  the  aircraft  comes  from  experience  and  practice.  Neither  the  current 
training  curricula  nor  classroom  time  affords  the  opportunities  to  develop  such  knowledge  in  the 
course  of  training. 

♦  Extensive  experience  and  practice  are  less  and  less  likely  on  the  newer  aircraft  (i.e.,  those  equipped 
with  BIT  capabilities).  Training  curricula  place  less  and  less  emphasis  on  delving  into  the  aircraft  as 
onboard  capabilities  such  as  BIT  proliferate. 

♦  Completion  of  a  flight  control  course  is  a  requirement  for  participating  in  an  impoundment  team.  This 
course  is  not  typically  offered  to  junior  or  relatively  inexperienced  maintainers,  but  only  to 
experienced  maintainers  and  supervisory  staff  either  as  (a)  refreshers  or  (b)  opportunities  for 
certification  to  work  on  impound  teams. 

■  The  scope  of  material  covered  in  the  on-base  flight  control  course  extends  from  practical  “theory” 
(as  embodied  in  the  TO)  to  standard  or  recommended  procedures.45 

■  Hands-on  experience  is  usually  limited  to  students  working  through  scripted  scenarios.  These 
scenarios  typically  involve  no  more  than  stepping  through  the  TO  in  relation  to  a  sample  problem. 

■  These  training  scenarios  do  not  usually  involve  giving  students  a  problem  that  they  must  figure 
out  on  their  own.  In  other  words,  the  training  scenarios  usually  begin  with  a  specification  of  the 
problem  up  front.  These  scenarios  usually  do  not  incorporate  elements  of  the  operational  context 
in  which  faults  occur.  These  “by  the  book”  stepwise  scenarios  do  not  force  the  students  to  explore 
the  systems  and  infer  what  the  problem  might  be.  As  such,  it  is  difficult  to  see  how  they  prepare  • 

45  The  standard  F-15  flight  control  curriculum  at  Nellis  consists  of  4  days  of  classroom  training,  1  day  of  training  on 

maintenance  BIT,  and  8  days  of  practical  education  working  on  a  maintenance  trainer  or  on  a  training  aircraft.  Flight 

control  students  may  sometimes  help  out  on  the  flightline,  but  that  is  not  a  prescribed  part  of  the  course. 
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the  student  for  diagnosis.  Because  they  do  not  involve  context-sensitive  fault  conditions,  these 
scenarios  would  not  seem  to  sensitize  the  student  to  real-world  faults. 

■  A  big  problem  affecting  new  trainees  is  that  out  on  the  real  world  flightline  there  is  rarely  enough 
time  or  resources  to  conduct  maintenance  activities  in  the  manner  in  which  they  were  trained. 

This  makes  for  a  significant  amount  of  adjustment  once  die  trainees  actually  get  to  the  flightline. 

4.1.16  Other  SME  Recommendations  for  information  Improvements 

Although  better  up-front  SA  on  in-flight  fault  context  was  the  obvious  top  choice  for  improved  maintains 
information,  it  was  not  die  only  thing  the  SMEs  cited  as  being  potentially  helpful.  Other  items  nominated 
for  their  “wish  list”  are  enumerated  and  discussed  in  the  following  subsections. 

4.1.16.1  Better  Access  to  Experiential  Info  within  the  Maintainer  Community 

In  spite  of  the  generally  voluminous  “official”  data  found  in  the  TOs,  there  is  much  relevant  and  useful 
information  that  can  only  be  obtained  from  maintenance  experience  with  a  particular  aircraft.  Such 
information  includes  tips,  tricks  of  the  trade,  and  illustrative  “lore”  derived  from  experience  with  difficult 
maintenance  problems.  Such  information  is  by  definition  inaccessible  across  units  and  to  newer/younger 
maintainers  unless  some  mechanism  for  recording  and  disseminating  it  is  developed.  Each  unit  logbook 
serves  as  the  primary  repository  for  such  experiential  knowledge.  The  SME  groups  uniformly  cited  the 
value  of  the  unit  logbooks  as  information  resources.  However,  the  logbook  entries  are  not  collated  or 
compiled  for  general  distribution,  and  they  are  “retired”  after  a  few  years.  This  means  that  the  “rearward 
horizon”  for  logbook  entries  (and  the  attendant  experiential  knowledge)  is  limited. 

The  SMEs  indicated  there  are  no  effective  online  channels  for  general  discussion  and  note-sharing  (e.g., 
chat  rooms;  bulletin  boards,  ListServ  forums).  The  only  established  channel  identified  for  dissemination 
of  such  experiential  knowledge  is  an  Eagle  Notes  newsletter  that  had  been  distributed  in  the  F- 15 
community;  although  there  was  some  question  as  to  whether  or  not  it  still  existed. 

4.1.16.2  Faster  or  Better  Access  to  Available  Onboard  Data 

One  of  the  ways  in  which  maintainers  can  obtain  better  situation  awareness  on  the  in-flight  discrepancies 
is  to  review  any  data  captured  and  recorded  during  the  flight.  The  existence,  scope,  volume,  and  format  of 
such  data  vary  from  aircraft  to  aircraft.  Those  aircraft,  which  offer  such  onboard  data  resources,  do  not 
always  provide  the  data  in  a  form  that  maintainers  can  readily  use.  Some  data  sets  have  to  be  downloaded 
and  sent  out  to  another  office  to  be  transcribed  and/or  translated  into  a  form  the  maintainers  can  access 
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can  access  and  use  on  the  flightline.  Time  spent  awaiting  such  processing  is  time  added  to  the 
maintenance  process  path. 

4.1.16.3  More  Detailed  Information  on  the  Interplay  Between  Specific  Parts 
and  Specific  Planes 

Modem  militaiy  aircraft  are  so  complex  that  some  tail  numbers  often  exhibit  their  own  peculiarities.  One 
relevant  aspect  of  this  “individualism”  is  that  a  particular  LRU  may  or  may  not  function  correctly  on  a 
given  aircraft.  This  dysfunctional  status  may  be  peculiar  to  the  particular  pairing  of  that  LRU  and  that 
aircraft  (i.e.,  the  LRU  may  work  just  fine  on  another  aircraft).  Multiple  fighter  SME  groups  (most 
especially  the  F- 15  and  F-16  SMEs)  noted  situations  in  which  an  LRU  that  failed  on  a  given  aircraft  was 
cycled  back  to  the  test  stations  only  to  test  good  there  and  end  up  back  on  the  flightline  in  another  aircraft 
where  it  worked  just  fine.  Without  making  provision  for  tracking  such  historical  incompatibilities, 
maintained  may  waste  time  swapping  in  an  LRU  already  known  to  not  work  on  a  given  aircraft. 

This  situation  is  distinct  from  that  in  which  generally  dysfunctional  parts  end  up  on  the  supply  shelf, 
although  that,  too,  was  an  issue  cited  by  most  of  the  SME  groups.  In  this  case,  an  LRU  repeatedly  goes 
back  and  forth  between  the  flightline  and  the  test  station,  possibly  working  for  a  few  sorties  in  the 
meantime.  In  most  units  these  bad  actor  LRUs  are  soon  identified,  tracked,  and  ultimately  shipped  for 
depot  repair;  although  frequently  not  before  they  have  caused  several  flight  discrepancies  and  possibly 
lost  sorties  as  well. 


4.1.16.4  A  Glimpse  at  the  Future:  Flight  Control  Maintenance  for  the  RQ-1 
and  F/A-22 

During  the  KA  visit  to  Nellis  the  team  was  offered  the  opportunity  to  interview  people  working  with  the 
RQ-1  Predator  UAV  and  the  newly  arrived  F/A-22  Raptor.  Because  of  the  potential  value  of  getting  a 
glimpse  at  the  future  context  for  flight  control  maintenance,  that  offer  was  accepted. 

4.1.16.4.1  The  F/A-22  Raptor 

One  of  the  commonly  cited  innovations  of  the  F/A-22  Raptor  program  was  its  approach  to  maintenance 
operations  and  diagnostics.  The  team  took  the  opportunity  to  gather  information  on  the  new  maintenance 
concept  of  operations  and  to  explore  what  it  signifies  in  terms  of  ongoing  maintenance  evolution.  The 
most  salient  general  points/issues  arising  in  discussions  with  the  Raptor  personnel  included  the  following: 
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♦  The  highly  automated  maintenance  concept  for  the  F/A-22  is  intended  to  accelerate  the  progress  from 
diagnosis  to  repair/replacement  action.  Intervening  steps  for  testing,  exploratory  troubleshooting,  etc., 
are  not  only  downplayed  but  intended  to  disappear  as  much  as  possible  from  the  maintenance  process. 
A  primary  goal  of  these  innovations  was  to  reduce  the  time  and  effort  invested  in  probing  the  aircraft 

to  diagnose  a  problem  (i.e.,  to  minimize  the  temporal  costs  of  the  troubleshooting  cycle  described  in 
Figure  1). 

♦  The  F/A-22  incorporates  a  high  degree  of  onboard  self-test  capabilities,  because  the  maintenance 
concept  emphasizes  the  aircraft’s  ability  to  self-test  so  as  to  efficiently  deliver  reliable  diagnoses  and 
action  recommendations  to  maintainers. 

♦  The  fault  isolation  elements  are  to  be  absorbed  into  IMIS.  The  concept  of  operations  is  to  interact 
electronically  with  the  aircraft  and  have  the  aircraft  provide  informative  responses/answers  to  probes. 
Troubleshooting  on  the  F/A-22  is  intended  to  be  done  via  software,  which  is  being  inserted  into  the 
maintenance  process  and  the  maintainers’  toolkit  like  never  before. 

♦  The  IMIS-based  diagnostic  software  will  interact  with  the  onboard  systems,  filtering  and  parsing 
along  a  virtual  fault  tree  to  vector  in  on  a  fault  condition.  The  end  point  of  this  diagnostic  process  is  to 
allow  the  combination  of  onboard  and  outboard  diagnostic  devices  to  tell  the  maintainer  what  part(s) 
need  to  be  replaced.  This  IMIS-driven  troubleshooting  concept  maximizes  the  role  of  the  automation 
and  minimizes  any  requirement  for  human  involvement  (in  terms  of  guiding  or  advising  the  course  of 
diagnosis). 

♦  The  F/A-22  includes  a  comprehensive  in-flight  data  capture  capability.  It  will  record  all  flight  data 
and  parameters  onto  a  Data  Transfer  Cartridge  (DTC).  This  data  can  then  be  downloaded  when  the 
aircraft  is  back  on  the  ground.  The  concept  is  intended  to  limit  pilot  reporting  involvement  to  only 
those  situations  in  which  out-of-range  values  or  conditions  are  encountered. 

♦  A  pilot  debriefing  will  still  be  conducted;  however,  the  pilot  will  not  have  to  rely  on  memory  and 
verbal  communications.  He/she  will  be  able  to  take  the  DTC  into  the  briefing,  where  the  data  will  be 
downloaded  into  IMIS.  Separately  from  the  pilot’s  debriefing  download  of  the  DTC,  a  “sortie 
download”  will  also  be  routinely  done. 

♦  The  F/A-22  maintainer’s  main  tool  will  be  the  Portable  Maintenance  Aid  (PMA)  -  a  portable/luggable 
computer  device  that  docks  to  the  aircraft  at  any  of  three  docking  ports.46 

♦  The  concept  is  that  maintainers  will  be  doing  the  sortie  download  with  the  PMA  in  parallel  with  the 
pilot  debriefing  using  the  DTC. 


46  Two  of  these  ports  are  located  in  each  of  the  main  wheel  wells  and  one  is  in  the  cockpit 
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♦  The  F/A-22  SMEs  concede  it  will  probably  take  years  to  get  this  operational  routine  to  the  point  it 
becomes  “routine.” 

♦  The  software-based  diagnostic  approach  in  the  Raptor  admittedly  does  not  account  for  physically- 
based  deficiencies  or  fault  sources  (e.g.,  wiring  faults  or  poor  connections). 

♦  The  only  fallback/reachback  position  for  F/A-22  maintained  at  this  early  stage  is  to  call  in 
engineering  technical  support.  There  is  a  significant  forward  deployment  of  contractor  technical  staff 
to  support  this  new  aircraft  at  Nellis. 

♦  The  goal  of  die  F/A-22  maintenance  concept  is  to  produce  a  completely  paperless  and  integrated 
maintenance  knowledge  base.  All  FIs  and  TOs  are  planned  to  be  incorporated  into  die  software 
support  suite.  No  provision  is  planned  for  providing  FIs  and  TOs  in  paper  form  (at  least  not  to  the 
frontline  maintained). 

♦  Once  the  on-site  maintenance  team  runs  through  the  technical  data  provided  by  the  current  systems, 
they  are  “done”  (i.e.,  there  is  nothing  more  they  are  supposed  to  do).  Once  they  reach  this  point,  the 
frontline  maintained  are  expected  to  call  in  the  technical  support  engineed.  This  immediate  and 
unavoidable  information  reachback  protocol  is  part  and  parcel  of  the  F/A-22  maintenance  concept, 
and  it  is  fully  expected  to  continue. 

♦  A  capability  for  the  F/A-22  to  transmit  real-time  in-flight  data  has  been  recommended,  but  it  has  not 
yet  been  funded.  The  issues  explaining  why  this  data  transmission  capability  is  not  in  progress 
apparently  have  more  to  do  with  security  concerns  than  funding  limitations. 

♦  The  SMEs  were  questioned  about  the  unusual  degree  of  reliance  the  F/A-22  concept  places  on  the 
PMA  devices.47  Their  responses  are  well  illustrated  by  one  SME’s  statement:  “The  conveniences 
outweigh  the  disadvantages.” 

♦  The  SMEs  concede  that  the  current  IMIS  data-/knowledge  base  has  been  assembled  on  a  site-specific 
and  sometimes  ad  hoc  basis.  Now  that  the  aircraft  is  being  used,  the  F/A-22  team  is  starting  to 
discover  instances  where  this  opportunistic  knowledge  base  assembly  has  left  gaps  and  variances. 

♦  Advantages  touted  for  the  PMA  included: 

■  Portability  +  integration  (i.e.,  everything  you  need  can  be  carried  in  one  package) 

■  Ready  access  to  complete  technical  data  at  the  aircraft 

■  Integrated  “one-stop”  access  to  forms,  TOs,  FIs,  and  other  supporting  documentation. 

■  The  windows  interface  provides  a  consistent  and  coherent  basis  for  user  navigation  and 
drilldown. 

♦  Disadvantages  or  problematical  issues  conceded  for  the  PMA  included: 


47  When  asked  if  the  F/A-22  could  be  effectively  denied  maintenance  support  if  the  PMA  devices  were  inaccessible  or  destroyed,  the  SMEs 
acknowledged  this  would  be  die  case. 
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■  The  software  is  currently  immature. 

■  The  knowledge  base  is  both  immature  and  not  all  that  refined  with  respect  to  the  parts  that  are 
already  in  place. 

■  The  PM  A  hardware  platform  was  state  of  the  art  around  1 0  years  ago  (when  it  was  introduced  in 
prototype  form),  but  it  is  showing  its  age  now. 

■  The  PMA  processor  is  relatively  slow  by  today’s  standards. 

■  Combined  with  the  complexity  of  the  software,  die  PMA  is  quite  sluggish  in  operation. 

■  There  is  no  touch  screen  capability. 

■  The  windows  interface  utilizes  relatively  few  graphics,  and  it  is  predominantly  text-based. 

4.1.16.4.2  The  RQ-1  Predator 

Unmanned  aerial  vehicles  (UAVs)  are  now  acknowledged  to  be  the  “wave  of  the  future.”  If  for  no  other 
reason  than  their  imminent  ubiquity,  the  team  believed  it  constructive  to  review  the 
RQ-1  Predator  to  see  what,  if  any,  points  could  be  discerned  about  flight  control  maintenance  for  this 
category  of  aircraft.  The  most  significant  points  obtained  in  this  session  included  die  following: 

♦  On  the  Predator,  all  flight  control  surfaces  are  operated  via  electromechanical  servomechanisms. 
Because  of  this,  all  flight  control  problems  on  the  RQ-1  are  “electronic  problems.” 

♦  Each  individual  flight  control  surface  can  be  isolated  and  independently  addressed  via  the  ground 
pilot’s  control  stick.  Control  inputs  are  interpolated  from  a  variable  voltage  flight  control  controller. 
These  inputs  comprise  die  digital  data  stream  outgoing  to  the  aircraft. 

♦  Flight  control  maintenance  on  the  Predator  is  definitely  distinct  from  such  maintenance  on  the  larger 
manned  aircraft.  Given  die  relative  technical  simplicity  of  the  Predator  airframe,  there  are  not  many 
details  to  troubleshoot.  On  the  other  hand,  troubleshooting  is  fairly  straightforward.  Even  on  this 
simple  airframe,  rigging  problems  do  occur.  These  are  most  commonly  evidenced  by  recurrent 
error/variance  in  a  single  flight  control  surface. 

♦  The  primary  guidance/computer  support  is  not  on  the  aircraft.  It  is  in  the  ground  control  station  (GCS) 
where  there  are  two  control  consoles  -  one  for  the  “pilot”  and  the  other  for  sensors  (controlling  and 
monitoring  the  reconnaissance  payload). 

♦  The  most  difficult  flight  control  issue  to  figure  out  is  discriminating  between  an  LRU  fault  and 
erroneous  data  attributable  to  the  GCS  or  the  communications  link. 

♦  The  communications  link  back  to  the  ground/pilot  station  can  be  a  source  of  errors  and  variances. 
Because  it  adds  another  potential  source  of  faults  and  glitches,  the  communication  link  represents  the 
downside  of  the  tradeoffs  in  using  UAVs. 
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♦  The  direct  communication  link  is  good  to  a  maximum  of  about  100  miles  with  line-of-sight.  Beyond 
this  range  control  has  to  be  routed  through  a  satellite  link.  Use  of  the  satellite  link  induces  a 
communications  delay  of  approximately  three  seconds. 

♦  On  the  ground,  troubleshooting  is  facilitated  by  hooking  the  aircraft  systems  directly  to  the  GCS. 
When  troubleshooting  on  the  ground,  one  can  switch  between  the  pilot  and  sensor  stations  to  cross¬ 
check  (a)  what  was  happening  with  one  side  when  an  event  was  occurring  with  the  other  and  (b)  to 
cross-check  interactions  between  the  two  aspects  during  a  mission,  maneuver,  etc. 

♦  In  effect,  the  RQ-1  has  its  “cockpit”  on  the  ground.  This  means  interactions  between  the  pilot  and  the 
aircraft  must  be  mediated  by  a  communication  link.  One  beneficial  outcome  is  that  there  must  be  a 
single  coherent  two-way  data  stream  carrying  all  aircraft  status  and  control  data  back  and  forth.  In 
other  words,  the  extent  of  captured  in-flight  data  aspired  to  by  the  other  maintenance  SMEs  is 
available  by  definition  with  the  UAV. 

♦  There  are  two  primary  modes  of  data  capture  for  a  Predator  mission:  an  8  mm  video  trace  and  the  data 
points  archived  from  the  digital  communication  link. 

♦  It  is  easy  to  swap  out  aircraft/GCS  pairs  to  crosscheck  interactivity  and  determine  which  of  the 
elements  is  the  problem.48 

♦  A  total  of  39  different  displays  or  screens  (termed  Variable  Information  Tables  -  VITs)  comprise  the 
ground  crew’s  interface  for  monitoring  the  RQ-1.  All  39  VITs  are  being  recorded.  This  means  that 
every  data  element  provided  die  ground  crew  is  being  archived  -  a  100%  flight  data  recording 
capability. 

♦  A  constant  data  sampling  rate  is  maintained  on  flight  controls  and  other  flight  data  during  a  mission. 
The  recording  medium  is  8mm  videotape  cassettes  with  an  effective  recording  time  of  about  two 
hours.  A  full  set  of  these  tapes  makes  for  a  comprehensive  mission  data  log. 

♦  Problem  duplication  is  a  matter  of  a  relatively  straightforward  process  of  elimination  with  the 
Predator.49 

♦  The  SMEs  stated  that  RQ-1  pilots  were  good  about  being  able  to  specify  when  and  in  what  context 
flight  control  problems  occurred.50 

♦  The  voluminous  data  that  is  recorded  is  in  numerical  format.  The  mathematical  software  package 
MatLab  can  be  used  to  translate  and  display  the  data  sets. 

48  This  may  not  sound  significant,  but  consider  that  it  is  die  equivalent  of  swapping  out  the  entire  cockpit  in  a  manned  aircraft 

49  This  was  illustrated  with  an  example  of  a  recent  flight  control  problem.  The  preceding  week,  a  flight  control  problem  had  been  reported  by  the 
pilot  The  bird  was  checked  against  one  known  to  be  flee  of  flight  control  defects.  It  turned  out  the  bird  checked  out  OK.  This  enabled  the 
maintained  to  quickly  zero  in  on  the  GCS.  They  determined  the  problem  resulted  from  an  out  of  tolerance  control  stick. 

♦  In  other  words,  the  sort  of  operator/pilot  information  the  other  SME  groups  rated  as  their  #1  desire  is  already  available  with  the  Predator.  One 
might  hypothesize  that  the  Predator  pilot  is  operating  with  less  situational  distraction  (G-forces,  visual  distractions)  than  a  pilot  inside  a  manned 
aircraft  This  would  reduce  relative  cognitive  burden  (from  distractions)  and  should  facilitate  real-time  SA  and  hence  post  hoc  memory  for 
problem  conditions  that  occurred. 
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♦  The  availability  of  so  much  relevant  data  allows  the  Predator  team  to  diagnose  relatively  subtle 
problems  in  short  order.  An  example  was  given  concerning  radio  frequency  interference  with  the 
flight  controls.  When  the  data  sets  were  reviewed,  they  could  readily  see  a  correlation  between  the 
observed  uncommanded  movements  and  the  triggering  of  certain  radio  equipment  on-board. 

♦  The  TOs  available  for  RQ- 1  diagnostic  support  are  “not  good.” 

♦  There  is  no  special  or  complicated  test  equipment  required  to  service  the  Predator. 

♦  At  the  present  time,  diagnosis  of  flight  control  problems  is  more  a  matter  of  experience  than  the  sort 
of  laborious  test-and-analyze  approach  used  on  the  manned  aircraft  that  were  reviewed. 

♦  The  only  tool  specifically  cited  as  used  for  flight  controls  was  an  electronic  inclinometer. 

♦  Instead  of  abstract  MFL  codes,  the  RQ-1  maintainers  deal  with  text  messages. 

♦  The  SMEs  gave  a  consensus  estimate  that  flight  control  problems  comprise  approximately  20%  of  all 
maintenance  discrepancies.  This  includes  the  tailboard  removal/reinstallation  that  must  be  done 
anytime  there  is  work  done  on  the  engine. 

♦  The  flight  control  servos  are  swapped  out  at  approximately  85  to  200  flight  hours.  Service  life  data  on 
the  servos  and  other  components  is  still  being  collated  and  analyzed.  It  is  rare  for  a  servo  to 
completely  fail. 

♦  The  documentation  requirements  for  RQ-1  maintenance  are  the  same  as  for  the  manned  aircraft.  Two 
sets  of  documentation  are  maintained  -  one  for  the  GCS  element  and  one  for  the  aircraft  itself. 

♦  Local/on-site  technical  support  is  readily  at  hand  and  very  deep. 

♦  In  the  event  of  a  flight  control  problem,  there  is  an  impound  procedure.  In  the  case  of  impoundment, 
there  are  two  impound  officials  -  one  for  the  GCS  and  one  for  the  aircraft. 

♦  The  SMEs  stated  their  biggest  maintenance  constraint  right  now  is  access/display/analysis  of  all  die 
data  they  have  available  to  them. 

♦  The  use  of  MatLab  to  access  the  logged  data  carries  an  overhead  for  mathematical  knowledge. 

♦  The  archived  data  constitutes  a  linear  trace  of  the  in-flight  parameters.  MatLab  turns  this  raw  data  into 
a  series  of  summary  line  graphs.  Analytical  interpretation  is  a  matter  of  visually  scanning  these  line 
graph  depictions  for  significant  indicators. 

♦  One  SME  who  had  worked  on  both  aircraft  flatly  stated  flight  control  fault  diagnosis  on  the  Predator 
was  “a  lot  faster  than  on  the  F-16.” 

♦  Data  is  captured  for  each  of  the  flight  control  servos.  This  means  it  is  relatively  easy  to  zero  in  on  the 
particular  servo  associated  with  a  reported  fault. 

♦  Rigging  tools  for  the  Predator  are  fairly  crude. 
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♦  One  big  advantage  of  the  UAV  setup  is  that  the  maintenance  person  can  be  called  in  to  literally  look 
over  the  pilot’s  shoulder  while  the  aircraft  is  in  flight.  As  one  of  the  SMEs  put  it,  this  “beats  any 
debrief.”  This  may  well  be  the  most  significant  point  obtained  with  respect  to  UAVs  and  their 
maintenance  process  path.  Not  only  can  the  pilot  notify  a  maintainer  of  a  flight  control  issue  while  the 
flight  is  still  in  progress,  the  maintainer  can  literally  be  briefed  on  the  problem  as  it  is  occurring. 
Combined  with  voluminous  in-flight  data  capture  and  the  ability  to  obtain  clues  on  flight  control 
issues  by  maneuvering  the  RQ-1  in  flight,  this  means  the  Predator  maintainer  has  access  to  a  degree 
of  situation  awareness  on  the  actual  problem  to  a  degree  that  other  maintenance  SMEs  can  only 
dream  of. 

♦  The  maintainer  and  the  pilot  can  jointly  perform  limited  in-flight  diagnostic  tasks  by  maneuvering  the 
aircraft  and  seeing  what  happens.  If  the  Predator  is  carrying  a  camera  package,  the  camera  can  even 
be  exploited  for  flight  control  diagnosis.  Within  limits,  it  can  be  rotated  to  allow  visual  inspection  of 
the  flight  control  surfaces  in  flight. 

♦  The  maintenance  reference  assets  are  both  new  and  problematical.  An  effort  was  only  recently 
undertaken  to  totally  re-do  the  Predator  TOs.  Predator  maintainers  do  not  use  FIs.  The  available 
information  support  is  peculiar  because  it  is  provided  in  a  Navy  diagnostic  reference  format. 

♦  One  particular  problem  is  the  flight  control  software  support.  Eveiything  about  the  pilot’s  control 
over  the  aircraft  is  software-mediated. 

♦  This  software  is  complex,  and  multiple  versions  may  be  in  use  at  any  given  time.  There  are  frequent 
changes/updates/upgrades  in  the  software.  All  this  makes  it  difficult  to  become  expert  with  the 
software,  because  it  is  always  a  moving  target. 

4.1.17  Summary:  Knowledge  Acquisition 

The  KA  visits  to  Nellis  and  Charleston  went  very  well,  and  the  team  obtained  considerable  data  in  a 
relatively  short  time.  The  attention  to  generating  a  KA  plan  (Appendix  C)  in  advance  of  the  trips  allowed 
the  team  to  coordinate  the  KA  team  effort  and  to  make  maximum  use  of  on-site  time.  The  “cascade 
approach”  to  incrementally  fleshing  out  team  understanding  of  flight  control  maintenance  atop  the  initial 
process  path  map  enabled  them  to  remain  organized.  Attention  to  gathering  data  on  auxiliary  units  of  the 
maintenance  organization  (i.e.,  test  stations;  support  sections)  and  currently  peripheral  information  on 
both  the  RQ-1  and  F/A-22  enabled  the  team  to  understand  the  core  maintenance  process  in  a  wider 
context.  The  data  presented  above  represents  only  a  portion  of  what  was  obtained.  Even  if  this  were  all 
that  was  gathered  on  the  two  KA  trips,  the  outcome  would  still  have  to  be  considered  a  solid  success. 
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4.2  Cognitive  Task  Analysis 

Cognitive  Task  Analysis  (CTA)  concerns  the  examination  and  critical  analysis  of  a  work  activity  or 
process  with  regard  to  the  cognitive  aspects  of  work.  Basically,  this  means  analyzing  the  “mental  work” 
associated  with  a  given  task  in  the  same  way  that  older  methodologies  analyzed  that  task’s  “physical 
work.”  A  task’s  “cognitive  aspects”  are  commonly  taken  to  include: 

♦  The  perceptual  acquisition  of  data  in  the  course  of  a  task 

♦  The  data  and  information  elements  critical  to  conducting  the  given  task 

♦  Mental  models  a  worker  employs  for  the  task  process  itself  and  the  subject  matter  he/she  must 
address  during  the  task 

♦  The  decisions  that  must  be  made  to  complete  the  task 

♦  The  critical  dimensions  of  decisions  made  in  the  task  (e.g.,  critical  data,  time  to  decide,  confounding 
factors,  mode  of  inference,  means  for  testing  alternatives) 

♦  The  degree  of  “cognitive  workload”  or  “cognitive  burden”  entailed  in  performing  the  task 

♦  Cognitive  and  informational  factors  which  can  induce  errors  and  other  degradations  in  task 
performance  (e.g.,  data  deficiencies,  data  overload) 

Maintenance  work  is  cognitively  intensive.  From  the  outset  the  maintainers  must  obtain  and  process 
potentially  large  amounts  of  data  (problem  reports,  symptoms,  diagnostic  data).  They  have  to  correlate 
this  data  with  their  available  models  (intemal/mental  and  extemal/diagrammatic  alike)  in  the  course  of 
generating  diagnostic  hypotheses  and  evaluating  both  these  hypotheses  and  the  course(s)  of  action  to  be 
undertaken.  There  are  decisions  to  be  made  on  a  range  of  topics  in  distinct  referential  contexts  (e.g., 
procedures,  functional  data,  hypotheses,  coordination  with  teammates,  aircraft  performance  and  viability 
for  duty,  etc.).  As  a  result,  maintenance  is  an  information-intensive  task,  and  cognitive  analysis  is  a 
significant  tool  in  understanding  this  task. 

In  die  following  sections  the  approach  to  collating  the  KA  results  into  a  coherent  cognitive  model  of  the 
maintenance  process  path  will  be  reviewed.  In  addition,  die  most  significant  points  derived  through 
analysis  will  be  discussed. 

4.2.1  Background:  Temporal  Orientation  to  Maintenance  Performance 
Assessment 

The  MXM  effort  was  geared  to  explore  the  interrelations  between  maintainers’  information  processes  and 
their  maintenance  tasks.  The  majority  of  the  points  cited  in  the  section  on  KA  relate  to  data,  information 
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resources,  and  process.  Applying  all  this  information  to  the  maintenance  task  requires  stating  what  it  is 
about  task  performance  one  is  seeking  to  analyze.  In  the  course  of  its  KA  work  the  team  obtained 
summary  statistics  on  USAF  maintenance  performance.  A  set  of  tabular  compilations  of  such  data  (for 
fighter  aircraft)  is  offered  in  Appendix  L  A  more  concise  summary  table  of  performance  statistics  for  four 
categories  of  fighter  aircraft  covered  in  the  KA  work  is  given  in  Table  6. 

Table  6:  CTA  Overview-Performance  Data  for  Four  Fighters  (FY93  •  FY02) 


4  HR  FIX 
RATE 

(Standard) 

4  HR  FIX 
RATE 

(Actual) 

8  HR  FIX 
RATE 

(Standard) 

8  HR  FIX 
RATE 

(Actual) 

A-10 

10-Yr  Average 

65.2 

67.5 

81 

82.57 

Shortfall  (10- 
Year  Average) 

+3.5%  (better 
than  standard) 

+1.9%  (better 
than  standard) 

10-Year  Trend 
(First  vs.  Last) 

Nominally 

Unchanged 

-7.9% 

-5.88% 

-6.4% 

F-15C/D 

10-Yr  Average 

57 

53.92 

75  (approx.) 

72.9 

Shortfall  (10- 
Year  Average) 

-5.4%  (short  of 
standard) 

-2.8%  (short  of 
standard) 

10-Year  Trend 
(First  vs.  Last) 

Unchanged 

-31.1% 

Nominally 

Unchanged 

-20.4% 

F-15E 

10-Yr  Average 

59.4 

53.6 

75  (approx.) 

73.2 

Shortfall  (10- 
Year  Average) 

-9.75%  (short  of 
standard) 

-2.24%  (short 
of  standard) 

10-Year  Trend 
(First  vs.  Last) 

+5.3% 

-25.8% 

Nominally 

Unchanged 

-18.5% 

F-16C/D 

10-Yr  Average 

66 

63.45 

85 

81.28 

Shortfall  (10- 
Year  Average) 

-3.86%  (short  of 
standard) 

-4.38%  (short 
of  standard) 
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4  HR  FIX 
RATE 

(Standard) 

4  HR  FIX 
RATE 

(Actual) 

8  HR  FIX 
RATE 

(Standard) 

8  HR  FIX 
RATE 

(Actual) 

10-Year  Trend 
(First  vs.  Last) 

Unchanged 

-20.1% 

Unchanged 

-14.13% 

Some  cursory  points  evident  in  Table  6  include  the  following: 

♦  In  absolute  terms,  4-  and  8-hour  fix  rate  performance  is  trending  worse  for  all  aircraft  over  the  10- 
year  period. 

♦  With  the  exception  of  the  A-10,  performance  results  fall  short  of  established  standards  for  the  10-year 
period  as  a  whole. 

♦  There’s  no  clear  pattern  to  the  summary  outcomes  with  respect  to  average  performance  (relative  to 
standard)  over  the  10-year  period.  For  half  the  aircraft  the  shortfall  for  4-hour  rate  performance  is 
worse  than  for  8-hour  performance,  while  the  reverse  is  the  case  for  the  other  half. 

♦  For  all  aircraft,  the  first-versus-last  year’s  downward  trend  over  the  ten  years  is  more  pronounced  for 
•  the  four-hour  fix  rate  than  for  the  eight-hour  fix  rate. 

Because  these  particular  statistics  are  framed  with  regard  to  time  (percentage  of  aircraft  fixed  in  a  given 
number  of  hours),  it  is  reasonable  to  characterize  the  evaluation  context  as  temporal.  The  fact  that  the  4- 
hour  rate  performance  has  worsened  farther  than  the  8-hour  rate  performance  is  consistent  with  a  situation 
in  which  die  maintenance  process  is  taking  longer  and  longer  to  effectively  complete.  Time  to  completion 
is  the  one  general  criterion  that  can  be  interpreted  to  subsume  negative  effects  resulting  from  a  variety  of 
possible  sources  (e.g.,  error,  grappling  with  ambiguity).  Given  its  generality  and  its  prominence  in  the 
available  evidence,  it  is  therefore  reasonable  to  adopt  temporal  maintenance  process  performance  as  the 
general  dimension  for  framing  the  analysis  of  the  maintenance  process. 

4.2.2  General  Observations  about  Temporal  Performance  Based  on 
Available  Statistical  Data 

It  is,  of  course,  very  risky  to  venture  conclusions  about  the  projects  focal  topic  (flight  control  maintenance 
in  particular)  on  the  basis  of  statistics  compiled  at  a  more  general  level  of  reference  (all  maintenance). 
Nonetheless,  there  are  some  broad  observations  that  can  be  offered  with  respect  to  the  informational 
aspects  of  maintenance  that  were  emphasized  in  this  study. 


68 


4.2.2.1  Observation  Regarding  Effects  of  BIT  Data  Capabilities 

If  BIT  capabilities  were  uniquely  influential  in  facilitating  the  overall  maintenance  process,  one  might 
well  suspect  that  maintainers  of  aircraft  with  BIT  capabilities  would  exhibit  better  temporal  performance 
data  than  those  lacking  them.  Conversely,  one  might  well  suspect  that  the  worst  temporal  performance 
would  be  exhibited  in  maintaining  aircraft  without  BIT  capabilities.  The  data  summarized  in  Table  6  does 
not  clearly  support  this  notion. 


4.2.2.2  Observation  Regarding  the  Effects  of  Automated  Diagnostic 
Decision  Aids 

If  automated  diagnostic  capabilities  were  uniquely  influential  in  facilitating  the  overall  maintenance 
process,  one  might  well  suspect  that  maintainers  of  aircraft  for  which  such  dynamic  inferential  support 
was  available  would  exhibit  better  temporal  performance  data  than  those  lacking  it  Conversely,  one 
might  well  suspect  that  the  worst  temporal  performance  would  be  exhibited  in  maintaining  aircraft 
without  such  aids.  The  data  summarized  in  Table  6  does  not  clearly  support  this  notion.  The  one  aircraft 
with  the  longest  standing  automated  diagnosis  aid  (the  F-16  with  EDNA)  is  not  the  one  with  the  best  4- 
hour  and  8-hour  fix  rate  performance  statistics. 

4.2.2.S  Observation  Regarding  the  Effects  of  Complexity  in  the  Aircraft  and 
Associated  Maintenance  Decision  Space 

It  is  reasonable  to  hypothesize  that  the  more  complex  the  aircraft  systems  are  the  more  complex  the 
associated  maintenance  decision  space  will  be.  It  is  also  reasonable  to  hypothesize  that  there  would  be  a 
negative  correlation  between  decision  space  complexity  and  the  expedience  with  which  it  can  be 
navigated  (and  hence  the  expedience  with  which  maintenance  processes  are  conducted).  Given  these 
notions,  one  might  well  suspect  that  the  least  complex  aircraft  systems  would  be  correlated  with  better 
temporal  maintenance  performance  statistics.  Unlike  the  former  two  notions,  this  one  is  consistent  with 
the  statistics  in  Table  6.  Of  the  listed  fighter  aircraft,  the  one  with  the  simplest  flight  control  systems  is  the 
A- 10.  As  the  figures  illustrate,  this  is  not  only  the  aircraft  with  the  “best”  overall  ten-year  performance 
numbers,  but  it’s  also  the  only  one  of  the  listed  aircraft  which  has  met  and  even  exceeded  its  maintenance 
performance  standards. 

As  stated  above,  these  are  merely  observations  on  the  available  data,  not  “proofs”  of  particular 
hypotheses.  If  anything  these  observations  pertain  not  to  what  can  be  proven  but  rather  what  cannot  be 
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proven  (at  least  readily  and  on  the  basis  of  this  evidence).  The  team  cannot  prove  that  BIT  capability  or 
the  application  of  automated  logic  improves  maintenance  performance.  By  the  same  token,  the  team 
cannot  disprove  that  complexity  in  the  subject  systems  (and  hence  in  the  associated  diagnostic  decision 
space)  degrades  maintenance  performance.  The  bottom  line  is  that  there  is  no  clear  evidence  indicating 
human  cognitive  performance  should  be  any  less  a  primary  concern  than  the  impacts  of  various 
technological  capabilities.  Having  said  that,  attention  will  now  turn  to  depicting  and  analyzing  such 
human  cognitive  factors  in  the  context  of  flight  control  maintenance. 

4.2.3  General  Points  Concerning  the  Maintenance  Process  Path 

The  following  points  can  be  made  about  the  maintenance  process  path  outlined  with  the  maintenance 
SMEs: 

♦  The  general  course  of  progress  through  the  process  path  is  not  random.  There  is  a  reliable  course 
through  the  process  path  that  is  followed.  One  does  not  start  or  proceed  with  just  any  step  in  the  path. 
As  a  result,  an  appropriate  model  of  the  process  path  should  account  for  the  general  linearity  of  a 
representative  instance  of  the  process  being  modeled. 

♦  This  general  course  is  not  invalidated  by  the  leaps,  cycles  or  truncations  noted  in  the  KA  sessions. 
The  fact  that  one  might  jump  ahead  or  backward  in  the  overall  path  doesn’t  invalidate  the  process 
model  because  these  leaps  are  transient  moves  in  what  remains  an  essentially  linear  line  of  progress. 
This  same  basic  principle  holds  for  cycles  and  truncations  as  well.  As  a  result,  an  appropriate  model 
for  the  process  path  should  not  be  incapable  of  accounting  for  such  optional  events. 

♦  The  most  telling  cumulative  performance  variable  in  evaluating  outcomes  of  this  process  path  is 
temporal  duration  (i.e.,  time  to  completion).  Measures  of  “quality”  or  “accuracy”  do  not  unilinearly 
accumulate  during  the  course  of  the  process  path.  At  any  subsequent  step,  measures  of  these  and 
similar  abstract  metrics  can  stagnate  or  even  reverse  (as  when  a  good  start  leads  to  a  misstep). 
Temporal  duration,  however,  accretes  linearly  and  irreversibly  throughout  the  course  of  progress 
through  the  process  path.  As  a  result,  an  appropriate  model  for  the  process  path  should  be  capable  of 
correlation  with  a  linear  timeline. 

♦  The  maintenance  process  path  entails  considerable  data  and  information  processing.  From  the  outset 
(in  receiving  a  pilot  report)  to  the  finale  (when  documentation  of  an  outcome  is  compiled,  presented, 
evaluated,  and  certified)  the  maintenance  process  is  an  information-intensive  activity.  As  a  result,  an 
appropriate  model  for  the  process  path  has  to  make  allowance  for  correlating  data  and  information 
with  other  elements. 

♦  However  information-intensive  it  is,  the  maintenance  process  path  cannot  be  comprehensively 
portrayed  solely  in  terms  of  information  processing.  Resolution  of  a  maintenance  problem  does  not 
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stop  with  an  abstract  diagnosis.  Tangible  action  is  required  to  affect  a  solution.  Along  the  way  to  this 
solution,  many  other  tangible  actions  (e.g.,  testing)  must  occur  in  progressing  through  the  process 
path.  As  a  result,  an  appropriate  model  for  the  process  path  has  to  make  allowance  for  correlating 
actions  with  other  elements. 

These  points  delineate  a  set  of  criteria  for  the  model(s)  that  can  appropriately  be  employed  to  usefully 
depict  the  maintenance  process  path.  In  the  following  section  a  model  meeting  these  criteria  will  be 
introduced  and  discussed. 

4.2.4  Selecting  a  Representational  Framework  for  the  MXM  CTA  Work 

There  are  a  number  of  models  and  frameworks  available  for  collating  and  analyzing  cognitive  task 
performance  issues.  Many  of  these  instruments  are  based  on  representation  and  analysis  of  one  worker 
performing  one  single  task.  Given  the  shift  of  focus  in  the  MXM  program  toward  coverage  across 
multiple  aircraft,  this  one-worker/one-task  arrangement  does  not  fit  the  purpose.  This  meant  the  team 
needed  to  select  a  representational  model  capable  of  generalization  across  specific  maintenance  processes 
(for  the  subject  aircraft)  and  capable  of  accounting  for  work  processes  undertaken  by  teams  (as  opposed 
to  individuals).  Second,  the  focus  on  the  application  of  data  and  information  in  the  maintenance  process 
constrained  the  range  of  cognitive  engineering  models  that  recommended  themselves  for  analytical 
purposes.  This  meant  that  a  representational  model  allowing  for  correlation  of  data  types  with  process 
steps  had  to  be  selected.  Finally,  maintenance  work  (although  cognitively  burdensome)  is  not  purely  a 
matter  of  information  processing.  It’s  not  unfair  to  characterize  maintenance  as  the  physical  manipulation 
of  a  malfunctioning  artifact  until  it  again  meets  expected  performance  criteria.  The  tangible  actions 
required  are  not  “informational,”  but  they  are  guided  and  evaluated  by  informational  (and  hence 
cognitive)  processes.  This  meant  that  a  representational  model  providing  a  basis  for  interrelating 
information  with  the  decisions  and  decided  courses  of  action  during  the  maintenance  process  path  needed 
to  be  selected. 

In  summary,  a  cognitive  model  was  needed  that  was  capable  of  usefully  representing: 

♦  The  overall  course  and  stepwise  sequencing  of  the  maintenance  process  path  (Figure  1 ) 

♦  The  practical  actions  or  courses  of  actions  undertaken  at  each  step  in  this  process  path 

♦  The  mapping  of  data  and  information  onto  this  path  sequence 

♦  The  manner  in  which  data  and  information  interrelate  with  decisions  made  and  actions  taken 
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The  two  cognitive  engineering  approaches  most  widely  invoked  at  this  time  each  have  deficiencies  with 
respect  to  these  selection  criteria.  The  means-ends  hierarchy  (sometimes  called  an  abstraction  hierarchy) 
developed  by  Jens  Rasmussen  (Rasmussen,  1986)  is  a  model  for  correlating  top-level  task  goals  to  low- 
level  activities  and  physical  implements.  This  model  yields  a  detailed  picture  of  the  “structural  elements” 
descriptive  of  a  given  task.  However,  this  model  is  a  static  “snapshot”  of  the  most  general  or  abstract 
elements  involved.  It  is  unable  to  depict  processual  elements  such  as  the  sequencing,  feed-forward,  and 
step-specific  information  requirements  that  the  team  sought  to  map  out.  Although  Rasmussen  does 
address  diagnostic  decision  paths  with  his  so-called  decision  ladder  model  (Ibid.),  this  model  does  not 
easily  correlate,  much  less  integrate,  with  the  detailed  abstraction  hierarchy. 

The  approach  labeled  naturalistic  decision  making  (NDM)  (Klein  et  al.,  1992)  emphasizes  critical 
incidents  and  indicators  associated  with  a  decision  maker  selecting  courses  of  action  under  conditions  of 
high  uncertainty  (e.g.,  battlefield  command  and  control).  This  approach  is  ill  suited  for  MXM  purposes 
for  multiple  reasons.  First,  its  focus  is  on  decision  making  in  and  of  itself.  Although  (as  mentioned  above) 
the  maintenance  process  path  is  laden  with  decisions  to  be  made,  it’s  not  fully  explainable  in  terms  of 
decisions  alone.  Second,  the  NDM  approach  is  geared  to  analyzing  situations  where  the  operational 
context  is  fluid  and  uncertain.  Although  this  is  perhaps  characteristic  of  the  early  diagnostic  phases  of  the 
maintenance  process  path,  it  is  not  characteristic  of  the  latter  phases  (in  which  the  maintainers  are 
woricing  in  the  context  of  the  very  deterministic  decision  space  of  “does  it  work  or  doesn’t  it?”).  Finally, 
the  typical  NDM  focus  on  individual  decision  makers  makes  this  approach  difficult  to  apply  to 
collaborative  team  situations  such  as  MXM  subject  matter. 

The  model  or  framework  judged  most  amenable  to  MXM  purposes  is  the  OODA  Loop  of  Col.  John  R. 
Boyd  (Boyd,  1987).  Boyd’s  OODA  Loop  has  become  a  dominant  analytical  device  in  the  0*1  literature, 
and  it  has  been  demonstrated  to  be  an  effective  framework  for  describing  and  analyzing  information 
operations  (Whitaker  &  Kuperman,  1996).  These  well-known  recent  applications  obscure  the  fact  that  the 
OODA  model  had  its  origins  in  explaining  the  perception-decision-response  sequences  entailed  in  high- 
performance  activities  for  individual  operators  (fighter  pilots). 

The  OODA  Loop  is  a  cyclical  interrelating  process  path  leading  from  perception  of  situational  data 
through  decision-making  and  on  to  resultant  action(s).  The  acronym  “OODA”  stands  for  Observation  - 
Orientation  -  Decision  -  Action,  and  the  “loop”  connotes  a  cyclic  iteration  through  these  four  steps 
(Figure  2). 


72 


OBSERVE 


Figure  2:  OODA  Loop  (per  Boyd,  1987) 

For  all  its  deceptive  simplicity,  Boyd’s  OODA  Loop  incorporates  several  key  features  that  make  it  useful 
in  cognitive  task  analysis.  First,  it  explicitly  frames  the  subject  matter  in  terms  of  continuous  process  from 
perception  (Observe)  through  cognition  (Orient/Decide)  to  response  (Act).  This  affords  a  scope  of 
referential  context  identical  to  that  of  MXM  (maintainers  perceiving  data  and  making  decisions  for 
ongoing  maintenance  actions).  This  permits  one  to  proceed  without  decomposing  the  subject  process  into 
multiple  (and  potentially  irreconcilable)  sub-models  (as,  e.g.,  would  have  been  the  case  in  applying 
Rasmussen’s  models).  Furthermore,  the  fact  that  the  OODA  model  encompasses  the  entire  path  from 
perception  to  action  affords  a  framework  for  correlating  relevant  elements  that  could  not  be  accounted  for 
in  other  models  (e.g.,  critical  incident  analysis). 

4.2.5  Mapping  the  maintenance  Process  Path  onto  an  OODA 
Representation 

Having  selected  a  representational  schema,  the  next  step  is  to  map  the  process  path  (Figure  1)  onto  this 
schema  and  populate  the  schema’s  constituent  “slots”  with  the  data  peculiar  to  the  process  being  so 
mapped.  For  the  purposes  of  this  report,  the  summary  results  of  this  mapping  are  presented  in  the  form  of 
1 1  tables  below.  The  reason  for  using  eleven  OODA  “maps”  for  the  eight-step  process  path  was  to  further 
break  down  the  troubleshooting  cycle  into  four  segments  so  as  to  both  (a)  account  for  the  subsidiary 
functions  subsumed  in  the  cycle  and  (b)  make  the  clearest  presentation  of  the  O-O-D-A  progression 
inherent  with  respect  to  each  of  these  subsidiary  components. 

Each  of  the  1 1  tables  provides  a  cursory  enumeration  of  elements  associated  with  each  of  the  four  OODA 
steps.  For  the  “Observe”  step,  these  will  be  elements  (e.g.,  data,  situations)  that  must  be  perceived.  For  the 
“Orient”  step,  these  will  be  elements  for  which  maintainers  must  achieve  situation  awareness.  For  the 
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“Decide”  step,  these  will  be  the  issues  or  topics  for  which  decisions  must  be  made.  Finally,  for  the  “Act” 
step,  these  will  be  the  typical  actions  (or  courses  of  action)  deriving  from  that  phase’s  Decide  step. 

As  applicable,  the  tables  also  include  a  listing  of  the  potential  “leaps”  (process  path  jumps  other  than 
simply  moving  forward  to  the  next  step)  that  might  occur.  The  set  of  “leaps”  noted  herein  reflect  both  (a) 
such  occurrences  specifically  mentioned  by  the  SMEs  and  (b)  other  such  jumps  discernible  in  the 
evidence  or  derivable  from  SME  accounts. 

4.2.5.1  Problem  Identification/Reporting 

This  is  the  initial  step  in  the  maintenance  process  path  identified  by  the  SMEs.  Using  a  brief  summary  set 
of  elements  derived  from  the  KA  materials,  this  step  can  be  mapped  onto  an  OODA  structure  as 
illustrated  in  Table  7. 


Table  7:  OODA  Map  for  Problem  Identification/Reporting 


Problem  Identification/Reporting 

Observe 

Orient 

Decide 

Act 

♦  Pilot  notification 

♦  Pilot  description  of 
problem/anomaiy 

♦  Indications  and 
warnings  (from  pilot; 
from  data  assets) 

♦  Determine  nature 
of  apparent  flight 
control  anomaly 

♦  Contextualize 
anomaly  with  in¬ 
flight 

conditions/actions 
(with  on-hand  data) 

♦  Discern  basic 
symptoms 

♦  Check  readily- 
accessible  A/C 
parameters  (e.g., 
switchology) 

♦  Viability/clarity  of 
problem  report 

♦  Viability/reliability 
of  reported 
symptoms 

♦  Viability  of  trying 
a  “quick  fix"  (e.g., 
reset) 

♦  A/C  subsystem(s) 
likely  to  be 
involved 

♦  Probe  for  more  info 
from  aircrew? 

♦  Switchology 
correction? 

♦  Reset  attempt? 

♦  Explain  why  anomaly 
doesn’t  require  fix 
(e.g.,  anomaly 
resulted  from 
inappropriate  in-flight 
action)? 

♦  VERY  QUICK  FIX? 

♦  OTHERWISE: 

Proceed  to  next 
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Potential  “Jumps”  in  Process  Path 

Forward 

Backward 

♦  Troubleshooting  Phase  1  (Setup)  •  Immediate 
capture  of  volatile  on-board  date 

♦  Troubleshooting  Phase  2  (Diagnostic  Hypothesis) 
for  either  future  reference  or  for  attempt  at  quick 
fix 

♦  Troubleshooting  Phase  3  (Acting  on  Diagnosis) 
for  “quick  fix” 

♦  Solution  Reporting/Documentation  (if  “quick  fix" 
works  and  gets  documented) 

♦  OUT  of  process  path  (if  immediate  “quick  fix” 
works  and  is  not  documented) 

OUT  of  process  path  if  there’s  a  determination  of 
no  actionable  problem 

The  basic  elements  to  be  observed  are  the  fact  and  content  of  the  aircrew  problem  report,  along  with  any 
auxiliary  data  or  clues  that  can  be  obtained.  At  this  point,  the  maintainer  needs  to  orient  to  the  state  of  the 
aircraft  and  any  available  data  to  set  an  initial  context  for  addressing  the  reported  problem.  Decisions  to 
be  made  at  this  early  point  primarily  relate  to  the  report  itself  and  (as  the  data  allows)  any  readily 
discernible  hypotheses  about  the  problem’s  cause.  Actions  in  this  phase  are  most  often  directed  toward 
fleshing  out  the  problem  report  and/or  making  an  immediate  resolution  response  (i.e.,  a  “quick  fix”). 
Maintainers  may  “leap”  forward  to  troubleshooting  if  confident  about  an  initial  hypothesis  or  even  to 
solution  reporting  if  a  quick  fix  works  (though  according  to  the  SMEs  quick  fixes  often  are  not 
documented).  In  the  event  a  “no-fix”  condition  is  recognized,  the  maintainer  may  “leap”  out  of  the 
maintenance  process  path  altogether  (i.e.,  he/she  is  “done”). 

4.2.5.2  Front-end  Unit  Coordination  (Getting  the  Aircraft  into  the 
Maintenance  Cycle) 

This  is  the  initial  step  beyond  perfunctory  “no-fix”  or  “quick  fix”  incidents  in  the  maintenance  process 
path.  This  step  can  be  mapped  onto  an  OODA  structure  as  illustrated  in  Table  8. 
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Table  8:  OODA  Map  for  Front-end  Unit  Coordination 


Unit  Coordination  (Get  A/C  into  Maintenance  Cycle) 

Observe 

Orient 

Decide 

Act 

♦  Unit  protocols  and 
procedural 
standards 

♦  Current  state  of 
personnel  and 
resources 

♦  Weather 

♦  Available  space 

♦  A/Cs  scheduled 
flight  itinerary 

♦  Roles  that  must 
be  represented  on 
impound  team 

♦  Prospects  for 
working  inside  vs. 
outside 

♦  Options  for 
placing  aircraft  for 
the  work 

♦  General  prospects 
for  the  aircraft’s 
return  to  service 

♦  Personnel  to  be 
assigned  to  team 

♦  Area  where 
maintenance  work 
will  be  conducted 

♦  Tentative 
prognosis  for  Pro 
Super 

♦  QA/QC 
coordination 
requirements 

♦  Advise  Pro  Super 
of  A/C  being 
removed  from 
duty  status 

♦  Assemble 
impound  team 

♦  Designate 

Impound  Official 

♦  Designate  Team 
Chief 

♦  Coordinate  with 
QA/QC 

♦  Document 
impound 

♦  Release  A/C  for 
maintenance  work 

At  this  point  the  maintainer’s  observations  are  made  with  regard  to  the  maintenance  organizational  and 
task  environments  (as  opposed  to  the  aircraft  per  se).  Similarly,  the  orientations  made  in  this  step  have 
more  to  do  with  organizational  context  than  the  specific  aircraft,  and  its  functionality.  Decisions  made  at 
this  point  relate  to  matters  surrounding  conduct  of  the  pending  maintenance  work  and  not  the  object  of 
that  work.  Finally,  the  actions  undertaken  relate  to  the  administrative  requirements  for  entering  the 
aircraft  into  the  maintenance  cycle.  Given  the  potential  complexities  of  the  organizational  space  in  which 
this  step  must  be  effected  (e.g.,  scheduling  conflicts;  organizational  rules;  documentation  requirements),  it 
is  as  cognitively  intensive  as  any  of  the  others. 


4.2.S.3  Maintenance  Setup/Preparation 

This  is  the  step  in  which  the  maintained  actually  begin  their  “maintenance  work”  per  se.  The  cognitive 
elements  of  this  step  fit  an  OODA  structure  as  illustrated  in  Table  9. 
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Table  9:  OODA  Map  for  maintenance  Work  Preparation 


Mx  Setup/Preparation 

Observe 

Orient 

Decide 

Act 

♦  A/C  location  and 
state 

♦  Available  space 
constraints 

♦  Personnel 
availability 

♦  Equipment 
availability 

♦  Initial  assessment 
of  work  to  be 
done 

♦  Prospects  for 
parking  A/C 

♦  Feasible 
coordination  of 
personnel  and 
equipment 

♦  Where  to  park 

A/C 

♦  AGE  required 

♦  Tools  & 
instruments 
required 

♦  Tech  data  (e.g., 
reference  aids) 
required 

♦  Maintenance 
agenda 

♦  Sourcing  for 
required  assets 

♦  Move  and  park 

A/C  for 

maintenance  work 

♦  Obtain  AGE, 
tools,  instruments, 
tech  data 

There  are  no  obvious  opportunities  for  leaping  either  forward  or  backward  in  the  process  path  in  this  step. 
The  maintenance  preparations  concern  the  ability  to  work  on  the  aircraft,  not  the  resolution  of  the 
reported  problem.  Because  such  resolution  is  by  this  point  the  exit  criterion  for  the  process  path,  nothing 
can  be  expected  (and  little  can  be  conceived)  to  occur  in  this  step  that  would  satisfy  that  criterion  and  end 
the  process. 


4.2.5.4  Troubleshooting  Phase  1:  Setup 

The  next  step  in  the  process  path  is  to  enter  the  troubleshooting  cycle  that  is  its  central  component  and  the 
type  of  activity  one  most  closely  associates  with  “maintenance.”  As  mentioned  earlier,  the 
troubleshooting  cycle  (Table  10)  has  been  subdivided  into  four  phases  to  facilitate  clarity  of  illustration  in 
applying  the  OODA  framework.  Phase  1  is  aimed  at  assembling  and  collating  the  initial  data  and 
information  relevant  to  diagnosis.  This  step  may  be  leapt  to  as  early  as  the  original  problem  report  - 
especially  if  immediate  action  is  required  to  capture  or  preserve  data  (e.g.,  volatile  digital  data  in  the 
aircraft’s  onboard  systems).  Backward  leaps  are  conceivable  in  two  cases.  The  first  and  most  likely  one  is 
leaping  back  to  the  problem  reporting  step  when  this  step  was  leapt  into  as  just  noted.  The  second,  less 
likely,  one  might  occur  if  something  in  the  data  being  assembled  at  this  point  indicates  a  need  to  back  up 
because  the  probable  root  fault  is  recognized  as  something  other  than  what  may  have  been 
extemporaneously  assumed  at  the  point  of  the  initial  problem  report. 
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Table  10:  OODA  Map  for  Troubleshooting  Cycle:  Phase  1 


Troubleshooting  Cycle 

Phase  1:  Setup 

Observe 

Orient 

Decide 

Act 

♦  Information  from 
Problem  ID  phase 

♦  Available  in-flight 
data  (e.g.,  pilot 
snapshots) 

♦  Availability  of  BIT 
capabilities  on 

A/C 

♦  Fault  data  (e.g.,  . 
MFL/BIT  codes) 
already  available 

♦  Volatile  data  at 
risk  of  being  lost 

♦  Diagnostic  data 
types  available 

♦  Transcription/ 
translation  reqt’s 
for  available  data 

♦  Salient  clues  from 
earlier  info 

♦  Prospects  for  A/C 
returning  to  duty 

♦  How  to  preserve 
volatile  data 

♦  Actions  necessary 
to  translate  data 
so  it  can  be 
mapped  onto 
diagnostic  aids 

♦  Agenda  for  data 
collation/ 
translation 

♦  External  support 
req'd  for  collation/ 
translation 

♦  Tentative 
guesstimate  for 
when  A/C  might 
return  to  duty 

♦  Download/capture 
volatile  data 

♦  Get  data 
translation  into 
motion 

♦  Receive/collate 
available 

diagnostic  data  as 
it  becomes 
available 

♦  Advise  Pro  Super 
of  tentative 
prospects  for  A/C 
return  to  duty 

Potential  “Jumps”  In  Process  Path 

Forward 

Backward 

NONE  -  Once  this  phase  is  entered,  the 
subsequent  phases  have  to  be  accounted  for 
(though  they  may  end  up  being  “run  through” 
perfunctorily) 

♦  Problem  ID/Reporting  if: 

■  Recognition  of  new/different  problem 

■  Volatile  data  captured  during  “leap- 

ahead” 

Phase  1  in  the  troubleshooting  cycle  would  appear  to  be  the  point  of  “commitment”  to  the  remainder  of 
the  process  path.  Once  you’ve  gotten  this  far,  it  will  be  incumbent  upon  you  to  follow  the  subsequent 
prescribed  steps,  even  if  only  in  a  perfunctory  fashion.  This  step  is  very  much  focused  on  data  in  and  of 
itself,  and  not  on  what  that  data  may  mean  for  diagnostic  and  prognostic  purposes.  This  step  is  distinct 
from  the  maintenance  Preparation  step  in  two  ways.  First,  this  step  primarily  addresses  data  compilation 
whereas  maintenance  Preparation  addresses  procedural  and  administrative  ramp-up.  Second,  the 
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maintenance  Preparation  step  is  more  generally  applicable  to  a  variety  or  set  of  similar  maintenance 
incidents,  whereas  this  step  is  particular  to  this  specific  incident. 


4.2.5.5  Troubleshooting  Phase  2:  Diagnostic  Hypothesis  Generation 

The  next  step  in  the  process  path  is  to  start  the  diagnostic  troubleshooting.  The  OODA  instantiation  of 
this  step  is  illustrated  in  Table  1 1. 


Table  11:  OODA  Map  for  Troubleshooting  Cycle:  Phase  2 


Troubleshooting  Cycle 

Phase  2:  Diagnostic  Hypothesis  Generation 

Observe 

Orient 

Decide 

Act 

♦  All 

data/information 
available  from 
earlier  phases 

♦  Data  incoming 
from  parallel 
translation 
support 

♦  Fault  data  (e.g., 
MFL/BIT  codes) 
already  available 

♦  Diagnostic 
reference  aid(s) 
such  as  FIs 

♦  Coding/indexing 
of  data  (e.g., 
specific  ID  of  each 
datum) 

♦  Correlate  data 
items  with 
reference  aid(s) 

♦  Layout/procedure 
s  for  using 
reference  aid(s) 

♦  Starting  point  in 
reference  aid 
structure  (e.g., 
fault  tree) 

♦  What  option(s)  or 
next  step(s)  the 
reference  aid 
indicates  for  the 
given  data 
item/code 

♦  Any  criteria  for 
discriminating 
among  multiple 
options 

♦  Any  test  action 
req'd  in 

conjunction  with 
proceeding  via  aid 

♦  What  next  step  to 
goto 

♦  Check  data 
against  reference 
aid 

♦  Sort  and  select 
among 

alternatives  (as 
req’d) 

♦  Proceed  stepwise 
through  (e.g.)  fault 
tree 

♦  Conduct  any 
auxiliary  tests  or 
data  checks  req’d 

♦  REPEAT  AS 
NECESSARY 
until  a  specific 
diagnosis  is 
obtained 
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Potential  “Jumps"  In  Process  Path 

Forward 

Backward 

Troubleshooting  Phase  4  (Closure)  if  problem 
determined  to  be  non-existent  or  not  actionable 

♦  Problem  ID/Reporting  if  there's  recognition 
of  new/different  problem 

♦  Maintenance  Setup  if  additional 
preparation/assets  are  identified  before  we 
can  proceed 

♦  Troubleshooting  Phase  1  (Setup)  if  new 
data/data  resource  invoked  or  new 
prognosis 

The  exploratoiy  process  of  testing  the  aircraft  and  trying  to  figure  out  where  a  problem  lies  is  at  the  core 
of  the  maintenance  process.  Success  in  conducting  this  phase  is  dependent  on  a  variety  of  factors,  and  it  is 
the  phase  in  which  the  maintained  are  most  dependent  on  the  data  and  information  assets  they  have  at 
their  disposal  (e.g.,  test  equipment,  the  TOs,  the  FIs,  their  background  knowledge,  experiential 
knowledge,  etc.). 

This  phase  is  particularly  problematical  in  the  case  of  flight  controls  (relative  to,  e.g.,  purely  avionics 
maintenance  work).  Although  the  specifics  vary  from  aircraft  to  aircraft,  “flight  control”  always  involves 
complicated  interactions  among  (e.g.)  electronic,  electrical,  hydraulic,  and  mechanical  subsystems.  A 
single  reported  anomaly  in  the  aircraft’s  in-flight  behavior  might  be  attributable  to  a  fault  in  one  or  more 
of  these  associated  subsystems.  This  means  that  unless  the  fault  is  clearly  (or  luckily)  isolated  on  the  first 
pass,  maintained  may  have  to  repetitively  test  and  evaluate  the  aircraft’s  various  relevant  subsystems 
before  determining  exactly  what’s  wrong.  Because  different  maintenance  expertise  may  be  required  to 
delve  into  one  or  another  subsystem,  this  exploratory  effort  may  entail  stopping  to  call  in  or  coordinate 
among  different  personnel. 

This  situation  is  complicated  by  the  fact  the  decision  space  (for  diagnosis  and  for  identifying  the 
subsystem(s)  underlying  the  fault)  is  not  subject  to  clear  and  straightforward  navigation.  For  example,  the 
relationship  between  electricals  and  mechanicals  involved  in  flight  control  is  not  all  that  deterministic. 
Granted,  it  is  usually  safe  to  say  the  electricals  trigger  or  control  the  mechanical  elements  that  in  turn 
control  the  flight  surfaces.  However,  this  doesn’t  mean  that  an  electrical  or  avionics  fault  that’s  been 
identified  rules  out  a  mechanical  aspect  to  the  problem.  Indeed,  the  coude  of  diagnostic  process  outlined 
by  the  various  SME  groups  consistently  started  with  avionics  and  moved  forward  (as  circumstances 
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warranted)  toward  the  mechanicals.  This  is  reflected  in  the  fact  that  the  FIs  include  occasional  formal 
instructions  to  stop  and  “check  rig”  (i.e.,  call  in  the  riggers  to  inspect  and  assess  the  mechanicals). 

This  doesn’t  mean  that  an  immediate  recognition  of  a  mechanical  fault  allows  the  maintenance  process  to 
skip  over  avionics  checks.  Identification  of  a  mechanical  problem  doesn’t  rule  out  an  electrical  or 
avionics  component  to  the  underlying  fault.  As  a  result,  it  can  happen  that  mechanical  troubleshooting  can 
lead  back  to  identifying  a  requirement  for  more  work  on  the  electricals  or  avionics.  This  typically  occurs 
through  process  of  elimination  rather  than  through  direct  implications  from  the  troubleshooting  per  se. 
Unlike  the  case  with  the  electrical-  or  avionics-oriented  FIs  typically  employed  up  front,  the  mechanical 
FIs  don’t  often  provide  instructions  to  call  in  the  avionics  people.  It  is  therefore  important  to  be  able  to 
make  a  strong  case  when  trying  to  kick  the  problem  back  into  the  electricals/avionics  maintainer’s  court. 
This  is  something  of  an  issue  because  the  riggers  can’t  readily  invoke  the  FIs  as  direct  evidence  that  such 
a  handoff  is  mandated  (in  contrast  with  the  opposite  direction  when  the  avionics  FIs  lead  to  “check  rig”). 

These  points  mean  that  even  during  this  exploratory  diagnostic  phase  the  maintainers  are  subject  to  time 
being  dedicated  to  team  and  administrative  coordination  issues  (as  opposed  to  the  diagnostic  work  itself). 

4.2.S.6  Troubleshooting  Phase  3:  Acting  on  Diagnostic  Hypothesis 

As  one  or  more  diagnostic  hypotheses  are  generated  in  Phase  2,  the  next  step  is  to  take  action  to  either 
verify  their  viability  or  to  affect  the  repairs)  they  recommend.  The  OODA  interpretation  of  this  step  is 
illustrated  in  Table  12. 
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Table  12:  OODA  Map  for  Troubleshooting  Cycle:  Phase  3 


Troubleshooting  Cycle 

Phase  3:  Acting  On  Diagnostic  Hypothesis 

Observe 

Orient 

Decide 

Act 

♦  All  data / 
information 
available 
from  earlier 
phases 

♦  Fault  data 
(e.g., 

MFL/BIT 

codes) 

already 

available 

♦  Diagnostic 
reference 
aid(s)  such 
as  FIs 

♦  Diagnosis 
arrived  at  in 
preceding 
steps 

♦  Implications 
of  current 
diagnosis 

♦  Implications 
of  preceding 
indications 
and  data 

♦  A/C 
(sub)systems 
involved  in 
current 
diagnostic 
implications 

♦  Whether  diagnosis  is 
consistent  with  reports, 
indications,  and  preceding 
data 

♦  IF  SO  -  What  repair  or 
replacement  actions  are  called 
for? 

♦  Does  it  turn  out  to  be  a  job  for 
AR  (rigging)? 

♦  IF  NOT  -  Should  we  proceed 
on  basis  of  current  best 
diagnosis? 

♦  IF  NOT  -  What  should  we  do 
to  backtrack  and  re-do  the 
diagnostic  process? 

♦  If  Backtracking  -  Do  we  stick 
with  the  diagnostic  aid  or  start 
exploratory  troubleshooting 
(e.g.,  tracing  wires)? 

♦  OTHERWISE -Is  it  time  to 
start  trying  “swapology”  to  see 
if  that  fixes  the  problem? 

♦  OTHERWISE -Is  it  time  to 
give  up  and  call  for  tech 
support? 

♦  If  Diagnosis 

Accepted  -  Proceed 
with 

repair/replacement 

action(s) 

♦  If  It’s  a  Rigging  Job  - 
Call  in  the  AR  folks 

♦  If  Diagnosis  Not 
Accepted  -  Backtrack 
to  explore  other 
possibilities 

♦  REPEAT  AS 
NECESSARY  until 
the  actionable 
implications  of  the 
current  diagnosis  are 
exhausted 

♦  OTHERWISE  -  Start 
with  “swapology” 
(replacing  parts  to 
see  if  that  fixes  the 
fault) 

OR 

♦  OTHERWISE  -  Make 
the  call  to  the  next 
available  layer  of 
tech  support  (e.g., 
AFETS,  contractor, 
depot) 
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Potential  “Jumps”  In  Process  Path 

Forward 

Backward 

NONE  -  Once  this  phase  is  entered, 
the  subsequent  phases  have  to  be 
accounted  for  (though  they  may  end 
up  being  “run  through”  perfunctorily) 

♦  Problem  ID/Reporting  if  there’s  recognition  of 
new/different  problem 

♦  Maintenance  Setup  if  additional  preparation/assets  are 
identified  before  we  can  proceed 

♦  Troubleshooting  Phase  1  (Setup)  if  new  data/data 
resource  invoked  or  new  prognosis 

♦  Troubleshooting  Phase  2  (Diagnostic  Hypothesis)  if 
results  (or  lack  thereof)  lead  us  in  another  direction 

This  Phase  3  and  the  preceding  Phase  2  comprise  a  possible  cycle  that  may  iterate  any  number  of  times. 
The  reason  for  separating  these  two  steps  in  this  OODA  layout  is  to  provide  segmentation  with  respect  to 
the  distinction  between  “theory”  versus  “practice.”  In  Phase  2  the  maintainers  are  addressing  the  current 
problem  with  respect  to  “theory,”  whereas  here  in  Phase  3  they  are  addressing  the  “practice” 
implementing  whatever  that  theory  (i.e.,  diagnostic  hypotheses)  recommends.  It  is  here  in  Phase  3  that 
something  is  done  in  accordance  with  the  current  diagnostic  hypothesis.  It  is  also  here  in  Phase  3  that  a 
decision  may  be  made  regarding  whether  to  either  (a)  start  hying  component  swapping  in  lieu  of 
specifically  prescribed  repairs  or  (b)  call  in  tech  support  in  the  absence  of  any  perceived  proper  action. 

4.2.5.7  Troubleshooting  Phase  4:  Reach  Closure  on  Current  Repair  Action 

At  some  point  in  the  troubleshooting  cycle,  the  maintainers  must  move  on  to  declaring  victory  and 
assessing  the  viability  of  having  done  so.  The  OODA  interpretation  of  this  step  is  illustrated  in  Table  13. 
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Table  13:  OODA  Map  for  Troubleshooting  Cycle:  Phase  4 


Troubleshooting  Cycle 

Phase  4:  Reach  Closure  On  Current  Repair  Action 

Observe 

Orient 

Decide 

Act 

♦  Diagnosis  arrived 
at  in  preceding 
steps 

♦  Completion  of 
repairs/ 
replacements 
prior  to  stopping 
point 

♦  Perceived 
assessment  of 
completion 

♦  Perceived  viability 
of  repairs/ 
replacements 

♦  Implications  of 
repair  actions  on 
mechanicals  and 
flight  surfaces 
(i.e.,  rigging 
matters) 

♦  Do  we  think  we’ve 
solved  the 
reported  problem? 

♦  Do  we  need  to 
call  in  the  riggers 
to  complete  the 
repair  action? 

♦  Call  in  riggers  if 
necessary  to 
complete  repair 
actions 

♦  Perform  final 
checks  to  ensure 
repairs  are 
completed. 

Potential  “Jumps”  In  Process  Path 

Forward 

Backward 

NONE  -  Once  this  phase  is  entered,  the 
subsequent  phases  have  to  be  accounted  for 
(though  they  may  end  up  being  “run  through" 
perfunctorily) 

♦  Problem  ID/Reporting  if  there’s  recognition 
of  new/different  problem  on  final  check 

♦  Troubleshooting  Phase  1  (Setup)  if  new 
data/data  resource  invoked  or  new 
prognosis 

♦  Troubleshooting  Phase  2  (Diagnostic 
Hypothesis)  if  results  (or  lack  thereof)  lead 
us  in  another  direction 

♦  Troubleshooting  Phase  3  (Acting  on 
Diagnosis)  if  final  checks  indicate  it’s  still 
not  fixed 

4.2.5.8  Solution  Reporting/Documentation 

Upon  completion  of  the  troubleshooting  cycle,  the  next  step  is  to  record  that  a  solution  has  been  achieved 
and  what  that  solution  turned  out  to  be.  In  terms  of  the  OODA  model,  this  step  can  be  depicted  as 
presented  in  Table  14.  At  this  point  the  context  begins  to  shift  back  toward  administrative  and 
organizational  elements  and  away  from  the  technical  features  which  were  the  foci  during  the 
troubleshooting  cycle. 
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Table  14:  OODA  Map  for  Solution  Reporting/Documentation 


Solution  Reporting/Documentation 

Observe 

Orient 

Decide 

Act 

♦  Fault  report, 
diagnosis,  and 
repair  actions 
from  preceding 
phases 

♦  Consensus  that 
repairs  now 
completed 

♦  Unit 

protocols/procedu 
res  for 
documenting 
repair  actions 

♦  Supply/parts 
reqt’s 

♦  Supply/requisition 
actions  to  date 

♦  The  course  and 
details  for  this 
maintenance 
action 

♦  Documentation 
reqfs  for 
completing  this 
action 

♦  Administrative 
reqfs 

♦  QA/QC  reqfs 

♦  Outstanding 
issues  with  supply 
chain  units 

♦  What  needs  to  be 
documented  to 
finish  off  this 
action? 

♦  Who  needs  to  be 
notified  of  the 
completion? 

♦  What  follow-up 
checks  are  or  will 
be  done 

♦  What  needs  to  be 
done  to  “settle  up” 
with  supply? 

♦  Advise  Pro  Super 

♦  Coordinate  with 
QA/QC  as  needed 

♦  Document  repair 
actions 

♦  Document  MIS 
assets  (e.g., 

CAMS) 

♦  Document 

Logbook 

♦  Complete  forms 
associated  with 
this  A/C  and  this 
maintenance 
event 

♦  “Settle  up”  with 
supply 

♦  Document  supply 
actions  taken 

4.2.5.9  Completion  Decision/Validation 

In  the  next  step  the  documented  solution  is  submitted  for  formal  review  and  certification  that  the 
maintenance  action  is  now  completed.  The  OODA  interpretation  of  this  step  is  illustrated  in  Table  15.  In 
this  step  the  context  of  attention  continues  its  shift  away  from  the  technical  toward  the  organizational  or 
administrative. 


85 


Table  15:  OODA  Map  for  Completion  Validation 


Solution  Validation/Verification  And  Completion  Decision 

Observe 

Orient 

Decide 

Act 

♦  Fault  report, 
diagnosis,  and 
repair  actions 
from  preceding 
phases 

♦  Consensus  that 
repairs  now 
completed 

♦  Components 
changed  out 
during 

maintenance 

process 

♦  Unit  protocols/ 
procedures  for 
documenting 
repair  actions 

♦  Supply/parts 
reqt’s 

♦  Supply/requisition 
actions  to  date 

♦  The  course  and 
details  for  this 
maintenance 
action 

♦  Final  consensus 
diagnosis 

♦  Action(s)  that 
fixed  the  reported 
problem 

♦  Set  of 
components 
swapped  out 

♦  Documentation 
reqf  s  for 
completing  this 
action 

♦  Administrative 
reqt’s 

♦  QA/QC  reqt's 

♦  Outstanding 
issues  with  supply 
chain  units 

♦  What  needs  to  be 
documented  to 
finish  off  this 
action? 

♦  Who  needs  to  be 
notified  of  the 
completion? 

♦  What  follow-up 
checks  are  or  will 
be  done 

♦  What  needs  to  be 
done  to  “settle  up” 
with  supply 

♦  Advise  Pro  Super 

♦  Coordinate  with 
QA/QC  as  needed 

♦  Document  repair 
actions 

♦  Document 
components 
swapped  out 

♦  Document  MIS 
assets  (e.g., 

CAMS) 

♦  Document 

Logbook 

♦  Complete  forms 
associated  with 
this  A/C  and  this 
maintenance 
event 

♦  “Settle  up”  with 
supply 

♦  Document  supply 
actions  taken 

♦  Documentation 
from  preceding 
phases 

♦  Impound  team 
composition 

♦  Unit  protocols/ 
procedures  for 
final  sign-off  on 
repairs 

♦  Availability  of 
maintenance 

Group 

Commander 

♦  Documentation 
reqt’s  for  sign-off 
briefing/meeting 

♦  Time  and  place 
for  meeting  with 
MXG  Commander 
for  final  sign-off 

♦  The  basic  “story 
line”  of  the 
maintenance 
process  to  be 
presented 

♦  Schedule  meeting 

♦  Conduct  meeting 

♦  Get  MXG 
Commander  sign- 
off  on  repair 

Potential “Jumps "  In  Process  Path 

Forward 

Backward 

None 

In  theory,  something  arising  at  this  point  might 
cause  the  team  to  fall  back  to  an  earlier  phase. 
None  of  the  SME  groups  mentioned  a  risk  of 
backtracking  once  this  phase  had  been  entered 
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4.2.5.10  Maintenance  Stand-Down 


Once  completion  of  the  current  maintenance  action  has  been  certified,  the  next  step  is  the  practical  matter 
of  cleaning  up  in  preparation  for  moving  on.  As  illustrated  in  Table  16,  this  is  neither  an  information-  nor 
a  decision-intensive  step  in  the  process  path. 


Table  16:  OODA  Map  for  maintenance  Stand-down 


Maintenance  Stand-down/Clea 

li-Up 

Observe 

Orient 

Decide 

Act 

Evidence  of  final  sign- 
off  for  this 
maintenance  action 

What  needs  to  be 
done  to  stand  down 

Course  of  action 
to  stand  down 

♦  Return  tools  and  instruments 
to  support  section 

♦  Perform  other  actions 
necessary  to  “clean  up" 

4.2.5.11  Back-End  Unit  Coordination  (Returning  the  Aircraft  to  Duty) 

The  final  step  in  the  overall  maintenance  process  path  is  to  get  the  aircraft  back  to  duty.  This  step  is 
illustrated  in  Table  17.  As  was  the  case  for  maintenance  stand-down,  this  is  neither  an  information-  nor  a 
decision-intensive  step  in  the  process  path. 


Table  17:  OODA  Map  for  Back-end  Unit  Coordination 


Unit  Coordination  (Return  A/C  To  Duty) 

Observe 

Orient 

Decide 

Act 

♦  Mission  itinerary 
for  newly-fixed 

A/C 

♦  Logistics  of 
getting  A/C  back 
to  duty  location 

♦  A/C  features 
requiring  setup  for 
next  mission 

♦  Towing/parking 
requirements 

♦  Any  mission- 
related 

modifications/ 
adjustments  to  be 
made 

♦  Mechanics  of 
getting  the  A/C 
back  to  duty 
location 

♦  Configure  A/C  for 
next  mission 

♦  Return  A/C  to 
duty  location 
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4.2.6  Critical  Cognitive  Issues  Discernible  in  the  KA  Results 

The  generation  of  a  structured  process  path  representation  is  a  central  goal  for  CTA  work.  However,  it  is 
not  the  only  desired  outcome  of  such  an  analysis.  One  other  goal  is  to  identify  critical  issues  that  affect 
the  cognitive  performance  (and  hence  task  performance)  of  the  subject  matter  experts.  In  this  section  the 
most  significant  such  cognitive  issues  identifiable  in  the  SME  groups’  comments  will  be  listed  and 
discussed. 

4.2.6.1  Issue:  Complexity  In  the  Problem  or  Decision  Space  for  Flight 
Control  Maintenance 

In  reviewing  the  aggregate  performance  statistics,  it  was  noted  that  the  negative  effect  of  complexity  was 
the  only  one  of  the  three  general  hypotheses  that  seemed  consistent  with  the  available  data  at  face  value. 
Because  “flight  control”  is  a  matter  of  interactions  among  electronic,  electrical,  hydraulic,  and  mechanical 
subsystems,  it  is  referentially  more  complex  than  most  other  objects  of  maintenance  (e.g.,  a  single  radar 
module).  An  in-flight  problem  might  well  derive  from  a  fault  in  one  or  more  of  these  interoperating 
subsystems.  This  means  that  in  troubleshooting  maintainers  might  have  to  repetitively  test  and  evaluate 
multiple  subsystems  before  isolating  the  precise  cause  of  the  reported  problem.  As  a  result,  the  decision 
space  or  problem  space  (abstract  set  of  elements  and  alternatives)  is  of  at  least  as  high  an  order  of 
complexity  than  is  the  case  for  most  other  aircraft  subsystems. 

Personnel  specializations  and  topical  foci  for  reference  aids  tend  to  reflect  this  subdivision  of  flight 
controls  into  distinct  domains  of  apparatus  to  be  collectively  considered.  This  makes  reference,  inference, 
and  progressive  navigation  difficult  (within  the  abstract  problem  space).  Unless  the  data  available  at  the 
outset  circumscribes  a  particular  component  or  class  of  components  as  likely  candidate  fault  sources, 
flight  control  maintainers  must  always  undertake  their  process  path  at  risk  of  having  to  grapple  with  these 
conceptual  complexities. 

4.2.6.2  Issue:  Criticality  of  Good  Up-front  Information 

For  the  sake  of  illustration,  consider  the  entire  maintenance  process  path  as  a  single  OODA  loop.  To  take 
a  final  action  requires  an  effective  decision.  This  in  turn  requires  adequate  and  accurate  orientation  to  the 
reported  problem  and  its  characteristics.  This  orientation  is  itself  contingent  upon  the  availability  of 
adequate  information  about  the  nature  of  the  fault.  In  other  words,  the  overall  inferential  process 
underlying  diagnosis  and  hence  repair  is  backward  dependent  on  how  much  the  maintainer  knows  (or  can 
know)  about  the  perceived  problem  at  the  very  beginning.  This  means  the  best  way  to  get  a  good  “head 
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start”  on  traversing  even  a  truncated  “quick  fix”  process  path  is  to  obtain  a  decent  picture  of  what  went 
wrong  in  this  particular  case.  As  a  result,  it  comes  as  no  surprise  the  #1  information  innovation  desired  by 
the  SMEs  was  better  descriptions  and  clues  at  the  point  the  problem  is  initially  reported. 

The  importance  of  good  up-front  problem  description  can  also  be  illustrated  in  terms  of  the  temporal 
dimension  emphasized  in  the  aggregate  performance  measures.  Deficiencies  in  initial  situation  awareness 
increase  the  amount  of  data  remaining  to  be  collected  and  reviewed  before  an  initial  hypothesis  can  be 
generated.  This  in  turn  increases  the  amount  of  effort  (and  hence  time)  required  to  accomplish  this 
additional  data  acquisition  and  analysis.  Increases  in  time  spent  at  this  early  stage  propagate  through  to 
the  final  cumulative  duration  of  die  given  maintenance  process.  In  addition,  to  the  extent  initial  SA  on  the 
reported  problem  aids  maintained  in  focusing  in  on  the  fault,  it  contributes  to  minimizing  time 
investments  as  the  process  moves  forward  into  and  through  the  troubleshooting  cycle. 

4.2.6.3  Issue:  Criticality  of  Simulating  or  Reproducing  the  Perceived 
Problem 

As  discussed  earlier,  the  #2  item  on  the  SME  groups’  wish  list  for  information  interventions  was  a 
capability  for  simulating  or  replicating  the  flight  control  problem  on  the  ground.  This  is  an  understandable 
backup  tactic  for  understanding  the  problem  in  the  absence  of  information  detailed  enough  to  immediately 
discern  a  fault’s  root  cause.  If  such  simulation  were  possible,  it  would  allow  maintained  to  replicate  and 
observe  anomalies  directly,  rather  than  being  forced  to  rely  on  whatever  information  they  could  obtain 
from  the  aircrew  and/or  available  in-flight  data.  Unfortunately,  this  is  difficult  or  impossible  to  do  in 
practice.  With  respect  to  cognitive  analysis,  the  most  important  aspect  of  this  #2  desire  is  not  its  potential 
for  implementation.  Instead,  it  is  important  for  the  fact  it  reinforces  the  theme  underlying  the  #1  wish  - 
i.e.,  a  need  to  better  understand  the  nature  of  the  problem  the  maintainer  is  being  asked  to  resolve. 

4.2.6A  Issue:  Fault  Sources  Unaccounted  for  in  Available  Diagnostic  Aid 
Representations 

Regardless  of  the  sophistication  attributable  to  available  diagnostic  aids,  all  the  embodied  models  of  the 
aircraft  share  something  in  common.  This  is  the  way  in  which  their  models  delineate  the  subject  matter  in 
terms  of  discrete  units  of  reference  (e.g.,  particular  LRUs).  Such  a  mode  of  reference  is  unavoidable; 
however,  it  necessarily  under-specifies  those  elements  of  the  subject  system  of  systems  that  lie  among 
these  unit  objects.  In  the  case  of  flight  control  systems,  such  interstitial  elements  include  wires, 
connectors,  valves,  hydraulic  lines,  etc. 
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The  KA  data  indicates  that  in  dealing  with  non-trivial  flight  control  problems  maintained  have  to 
dedicate  considerable  time  and  effort  to  addressing  such  interstitial  elements.  “Shooting  the  wires”  is 
something  the  F- 15  and  F-16  maintainers  repeatedly  alluded  to  as  a  troublesome  task.  The  relative 
importance  of  interstitial  components  as  candidate  sources  of  fault  is  well  illustrated  in  the  uniform  and 
universal  comments  made  regarding  connections  and  connectors  as  “culprits.”  Understanding  the 
underlying  linkages  and  relationships  among  aircraft  subsystems  -  one  of  the  key  characteristics  attributed 
to  expert  maintainer  knowledge  -  can  be  seen  in  terms  of  understanding  “what  lies  among”  the  discrete 
units  most  commonly  referenced  in  training  and  documentation. 

Does  this  mean  that  diagnostic  aids  can  and  should  be  generated  to  deal  with  all  the  interstitial  elements 
once  and  for  all?  No,  because  there  will  always  be  “interstices”  among  whatever  set  of  referents  is 
invoked  to  model  the  subject  system  and  depict  it  to  an  maintenance  user.  Instead,  it  means  that 
accounting  for  interstitial  elements  will  always  entail  knowledge  derived  from  dealing  with  the 
interrelationships  among  components  -  i.e.,  the  veiy  sort  of  experiential  knowledge  which  typifies  experts 
(relative  to  novices)  and  which  goes  relatively  undocumented  and  unshared. 

4.2.6.S  Issue:  Progressive  Deskilling  in  the  Maintainer  Population 

In  the  course  of  the  KA  activities  the  team  probed  for  perceived  distinctions  between  expert  and  novice 
abilities.  Naturally,  such  distinctions  are  to  be  expected  in  any  task  environment.  In  this  case,  however, 
the  SMEs  repeatedly  noted  states  or  trends  with  regard  to  the  relationship  between  expert  and  novice 
capabilities  that  are  causes  for  concern.  It  is  no  surprise  to  hear  that  novices  are  neither  as  proficient  at 
diagnostic  troubleshooting  nor  as  knowledgeable  about  a  subject  aircraft  as  their  more  experienced 
colleagues.  It  is,  however,  a  disturbing  surprise  to  hear  an  apparent  consensus  (among  trainers  and 
experienced  maintainers)  that  these  relative  disadvantages  are  trending  both  (a)  more  common  among  the 
maintainer  population  and  (b)  more  permanent  as  features  of  newer  maintainers’  careers. 

First,  there  were  consistent  (senior/experienced)  SME  comments  critical  of  the  general  technical 
knowledge  evidenced  by  younger  or  novice  maintainers.  It  is  not  just  that  younger  maintainers  do  not 
know  how  this  aircraft  functions,  the  problem  is  that  they  display  little  understanding  of  how  any  aircraft 
functions.  For  example,  the  trainer  SMEs  at  both  Nellis  and  Charleston  consistently  claimed  new  staffers 
emerge  from  technical  school  with  less  basic  technical  understanding  than  was  once  the  case.  This  puts 
newer  maintainers  at  a  disadvantage  in  being  able  to  undertake  free-form  exploratory  troubleshooting 
once  they’ve  exhausted  the  standard  diagnostic  guides  (e.g.,  the  FIs). 
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Second,  there’s  a  decreasing  potential  for  the  younger  maintainers  to  acquire  such  technical  knowledge  in 
the  course  of  their  work.  As  time  goes  on,  maintenance  becomes  geared  more  and  more  to  divining  faults 
on  the  basis  of  given  BIT  codes  and  the  canned  logic  of  fault  trees.  Furthermore,  the  trend  toward 
modularizing  the  subject  matter  into  LRUs,  which  are  swapped  out  rather  than  dissected,  diminishes  the 
opportunities  for  novices  to  explore  and  learn  about  how  the  aircraft’s  internal  components  operate. 

Finally,  and  most  disturbingly,  the  above-cited  factors  are  claimed  to  have  engendered  a  growing  reliance 
on  whatever  troubleshooting  guidance  is  made  available  in  canned  form,  whatever  courses  of  action  can 
result  from  procedures  designed  and  trained  in  strict  accordance  with  simply  following  that  canned 
guidance,  and  whatever  repairs  can  be  effected  by  changing  out  LRUs.  The  often-cited  ability  of  experts 
to  “dig  into”  the  aircraft  after  die  easy  solutions  have  been  ruled  out  is  based  on  knowledge  and  skill.  The 
novices  are  in  effect  denied  this  sort  of  knowledge  and  skill  as  time  goes  on.  In  other  words,  newer  or 
younger  maintainers  are  subject  to  progressive  “deskilling”  as  time  goes  on. 

This  increasing  proportion  of  relatively  deskilled  workers  cannot  help  but  result  in  diminishing 
performance  for  the  workforce  as  a  whole.  Multiple  examples  of  such  performance-degrading  effects 
were  cited  by  the  SMEs.  Novices  are  more  likely  to  call  in  technical  support  personnel  when  they  exhaust 
their  diagnostic  cookbooks.  This  necessarily  adds  external  coordination  costs  (in  time  and  effort)  to  the 
maintenance  effort.  Novices  can’t  very  well  be  expected  to  dig  into  the  aircraft’s  internals  when  they 
haven’t  been  trained  to  do  so.  In  any  case  they  are  claimed  to  lack  the  technical  knowledge  to  support 
such  courses  of  action.  Their  reliance  on  canned  diagnostic  logic  makes  them  vulnerable  to  each  and 
every  gap  or  deficiency  within  that  logic  (and  such  gaps  and  deficiencies  were  noted  by  all  SME  groups). 

4.2.6.6  Issue:  Value  of  Experiential  Knowledge  to  the  Maintainers 

There  is  much  relevant  and  useful  information  that  can  only  be  obtained  from  maintenance  experiences 
with  particular  aircraft  in  specific  instances.  Such  information  includes  tips,  tricks  of  the  trade,  and 
illustrative  “lore”  derived  from  difficult  cases.  The  SMEs  uniformly  alluded  to  such  experiential 
knowledge  in  terms  of  the  following  points: 

♦  Experiential  knowledge  is  a  key  discriminator  between  expert  and  novice  maintainers. 

♦  Experiential  knowledge  can  often  prove  a  valuable  asset  in  handling  tough  cases  -  particularly  once 
the  official  diagnostic  logic  is  exhausted. 

♦  Experiential  knowledge  is  shared  within  a  maintenance  unit  only  to  the  extent  it  propagates  via 
personal  interactions  and  whatever  portion  gets  entered  into  the  logbook. 
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♦  Generally  speaking,  there  are  no  formal  means  for  disseminating  such  experiential  knowledge  among 
units. 

♦  The  current  training  curricula  and  training  timeframes  do  not  permit  such  detailed  experiential 
knowledge  to  be  imparted  to  trainees. 

Such  experiential  knowledge  is  therefore  a  key  component  of  the  knowledge  base  the  maintainers  exploit 
in  the  conduct  of  their  work.  The  fact  that  this  key  component  is  under-supported  by  formal  procedures 
and  support  tools  constitutes  a  significant  cognition-related  deficiency  in  the  way  maintenance  work  is 
accomplished  today. 

4.2.7  Summary:  Cognitive  Task  Analysis 

The  data  obtained  during  the  KA  effort  has  been  applied  to  generate  a  coherent  process  path  map  in 
accordance  with  a  model  selected  as  best  suited  to  this  project’s  goals.  This  process  path  map  has  been 
leveraged  to  illustrate  the  general  linkages  between  data/information,  decisions,  and  actions  at  each  step 
in  the  process  path.  Finally,  file  most  important  subset  of  the  cognitive  issues  identified  from  the  KA  has 
been  reviewed. 


4.3  Information  Requirements  Analysis 

Information  Requirements  Analysis  (IRA)  concerns  the  examination  and  critical  analysis  of  a  work 
activity  or  process  with  regard  to: 

♦  The  set  of  data  and  information  required  to  perform  the  given  task 

♦  The  set  of  data  and  information  typically  available  to  the  worker  during  a  task 

♦  The  differences  between  these  two  sets  and  their  implications  for  improving  task  support 

In  the  knowledge  acquisition  section  the  team  reviewed  the  key  issues  and  questions  the  SMEs  cited  as 
important  in  pursuing  the  flight  controls  maintenance  process  path.  Each  of  these  items  reflects  or 
recommends  a  data  element  that  should  be  available  for  maintainers  to  exploit.  In  the  section  on  cognitive 
task  analysis  the  types  of  data  or  information  in  the  context  of  an  ordered  OODA  representational  schema 
were  reviewed.  This  mapping  correlated  the  issues  and  questions  obtained  during  KA  with  a  structured 
model  of  the  maintenance  work  being  studied. 
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4.3.1  Information  “Reachback”  in  the  Maintenance  Process  Path 

One  of  the  most  salient  characteristics  of  the  maintenance  process  is  that  it  is  collaborative  in  that  it 
involves  multiple  players  jointly  working  to  diagnose  and  repair  the  reported  fault.  It  is  not  the  case  that 
all  possible  players  participate  in  all  reported  cases.  Some  sources  of  data  and  expert  knowledge 
participate  only  if  invoked  by  the  front  line  maintainers.  This  constitutes  a  discretionaiy  reachback  slfor 
relevant  information  as  circumstances  warrant. 

Although  the  precise  details  vary  from  aircraft  to  aircraft,  the  SMEs  outlined  three  representative  steps  in 
a  progression  of  information  reachback.  The  first  is  to  on-base  tech  support  personnel  (e.g.,  AFETS, 
forward-deployed  contractors).  The  second  is  to  remote  (off-base)  technical  expertise  accessible  via 
telecommunications  (e.g.,  hot  line).  The  third  is  to  technical  expertise  accessible  by  calling  in  personnel 
stationed  at  the  relevant  depot  (e.g.,  “depot  assist”).  Each  of  these  reachback  assets  is  in  possession  of 
technical  data  resources  exceeding  the  scope  and/or  depth  of  those  available  on  the  flight  line.  The 
general  form  of  this  reachback  progression  is  illustrated  in  Figure  3. 

In  general,  maintainers  attempt  to  complete  the  maintenance  process  path  using  their  available 
information  assets  on-site.  It  is  only  when  they  reach  a  perceived  impasse  that  they  begin  to  exercise  their 
options  for  information  reachback.52  The  first  line  of  recourse  is  on-base  tech  support.  Where  available, 
the  second  line  consists  of  remote  tech  support  that  can  be  accessed  via  telephone.  The  third  line  of 
recourse  is  to  call  in  support  personnel  from  the  depot. 


91  Use  of  the  term  “reachback”  has  become  widespread  during  the  1990s.  However,  its  connotations  are  not  precisely  the  same  in  all  USAF 
communities.  For  example,  in  the  logistics  community  “reachback”  is  used  to  generally  refer  to  supply  transactions  in  support  of  front  line 
operations.  The  use  of  the  term  “reachback”  in  this  section  is  based  on  the  usage  of  that  term  in  information  operations  (10)  -  i.e.,  the  concept  of  a 
warfighter  (or  other  operator)  having  the  means  to  “reach  back”  to  rearward  support  elements  to  obtain  data  or  information  as  needed  to  support 
decisions  and  actions  in  the  present  task  or  situation.  In  this  usage,  “reachback”  connotes  the  front  line  operator’s  capacity  for  proactively 
initiating  demand-pull  transactions  (i.e.,  the  “supply  side”  component  of  the  objective  of  “getting  the  right  information  to  the  right  warfighter  at 
the  right  time”) 

32  The  exact  course  of  immediate  reachback  will,  of  course,  vary  with  circumstances  and  exact  location.  Comments  from  the  SME  groups 
indicated  that  on-base  tech  support  personnel  are  more  or  less  accessible  or  proactively  involved  in  checking  on  maintenance  actions  in  progress. 
For  example,  F-16  SMEs  at  Nellis  seemed  to  indicate  their  AFETS  technical  staff  are  more  readily  at  hand  than  the  C-l  7  tech  support  staff  at 
Charleston,  who  seemed  to  remain  in  the  background  until  and  unless  actively  summoned.  Another  source  of  variation  derives  from  whether  or 
not  manufacturer/contractor  support  staff  happen  to  be  deployed  on  base  (as  is  die  case  with  die  Boeing  support  staff  at  Charleston). 
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Figure  3:  General  Outline  for  Diagnostic  Reachback 


94 


4.3.2  Information  Reachback  as  a  Negative  influence  on  Overall 
Maintenance  Performance 

With  regard  to  maintenance  performance,  the  key  point  is  that  each  reachback  action  serves  as  a  drag  on 
maintainers’  ability  to  resolve  the  reported  maintenance  problem  in  a  given  timeframe.  More  specifically, 
reachback  degrades  temporal  performance  metrics  because: 

♦  Setting  aside  the  maintenance  activity  to  call  in  outside  help  may  involve  practical  actions  (e.g., 
parking,  putting  away  tools,  etc.)  which  either  consume  time  or  mandate  time  consumption  for  getting 
back  on  task. 

♦  Some  amount  of  time  will  have  to  be  consumed  in  making  effective  contact  with  the  next  reachback 
resource. 

♦  Some  amount  of  time  will  have  to  be  consumed  in  relating  the  maintenance  problem  and  the  work  so 
far  to  the  reachback  resource. 

♦  Where  personal  contact  on-site  is  needed,  there  will  be  time  consumed  in  coordinating  (e.g.)  travel, 
meeting  times,  etc. 

♦  Where  personal  contact  is  needed,  there  may  be  redundant  time  consumption  for  briefing  the 
technical  support  staff  once  they  arrive. 

These  general  points  pertain  whether  the  reachback  asset  is  on  base  or  at  a  remote  location.  The  main 
point  is  that  each  reachback  action  consumes  time  and  therefore  lengthens  the  end-to-end  duration  of  the 
given  maintenance  process. 

4.3.3  Primary  Information  Needs  Identified  in  the  KA  and  CTA  Efforts 

Given  the  discussions  above  and  in  the  preceding  sections,  the  main  information  deficiencies  (and  hence 
information  needs)  for  flight  control  maintenance  staff  can  be  summarily  listed  as  follows: 

♦  Detailed  and  comprehensive  description  of  the  perceived  problem  as  it  was  encountered  during  flight 
(the  SMEs’  #1  wish  list  item). 

♦  Access  to  experiential  knowledge  to  fill  in  the  gaps  in  the  formal  information  base. 

♦  Capture  and  dissemination  of  experiential  knowledge  to  help  bring  less  experienced  maintainers  up  to 
more  expert  levels  of  performance. 

♦  Dissemination  of  experiential  knowledge  across  maintenance  units  to  minimize  the  need  to  “reinvent 
the  wheel”  in  developing  tips  and  tricks  of  the  trade. 

♦  Attention  to  real-world  operational  context  in  training  materials  and  courses. 

♦  Better  background  technical  knowledge  for  incoming  trainees. 
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♦  Better  knowledge  of  an  aircraft’s  inner  workings  to  provide  less  experienced  maintainers  with  a 
foundation  for  analyzing  the  data  their  reference  and  diagnostic  aids  provide  them. 

4.3.4  Information  Capacities  and  Requirements  for  Addressing  SMEs’ 
#1  Desire  for  Improvement 

As  discussed  earlier,  the  SME  groups  clearly  and  uniformly  cited  better  situation  awareness  on  in-flight 
fault  context  as  their  #1  desired  improvement.  Modem  military  aircraft  are  complex  systems,  and  their 
flight  control  mechanisms  are  complicated  aggregates  of  avionics,  hydraulics,  and  mechanical 
components.  As  a  result,  the  decision  space  for  diagnosis  is  relatively  complex.  Any  and  all  clues 
available  up  front  allow  the  maintained)  to  more  expeditiously  vector  in  on  a  likely  diagnosis  and 
proceed  toward  resolution  or  repair.  The  most  straightforward  approach  to  maximizing  such  up-front  SA 
is  to  capture  and  record  as  much  in-flight  data  as  possible. 

With  some  aircraft,  the  maintenance  staff  already  enjoys  a  relative  abundance  of  such  data,  while  for 
other  aircraft  they  have  to  make  do  with  what  little  they  can  get.  The  maintainer  SMEs  who  seemed  most 
content  with  their  current  or  prospective  in-flight  data  access  were  those  working  with  the  C-17,  the  RQ-1 
Predator  UAV,  and  the  F/A-22  Raptor.  The  C-17  is  a  relatively  new  aircraft  (compared  to  the  fighters 
surveyed),  and  its  onboard  computer  systems  afford  a  good  capability  for  capturing  data  in  a  form  that  can 
be  downloaded  and  made  available  for  maintainer  reference  and  analysis.  Assuming  its  highly  centralized 
and  automated  systems  work  as  advertised,  the  F/A-22  promises  to  offer  maintainers  a  degree  of  insight 
into  in-flight  events  and  parameters  unparalleled  in  any  previous  fighter.  The  unexpected  surprise  is  the 
situation  enjoyed  by  the  UAV  maintainers,  who  have  it  best  of  all.  Not  only  are  they  afforded  a 
comprehensive  data  set,  they  also  have  the  ability  to  consult  with  the  pilot  and  jointly  examine  flight 
behavior  while  the  flight  is  still  in  progress. 

Multiple  SMEs  involved  with  the  older  fighters  touted  the  utility  of  data  “snapshots”  (pilot-triggered 
recordings  of  selected  system  data).  The  A- 10  SMEs  claimed  their  snapshot  access  was  a  big  help  in 
analyzing  reported  problems.  The  older  fighters’  maintenance  SMEs  at  Nellis  all  made  envious  reference 
to  the  CDDS  system  installed  on  the  B-1B,  which  allows  a  pilot  to  capture  a  voluminous  snapshot  of 
current  data  and  parameters,  including  cockpit  “switchology.”  The  F-l  5E  maintainers  were  happy  to 

report  that  a  snapshot  capability  was  finally  being  made  available  on  this  most  recent  edition  of  their 
aircraft. 
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Though  certainly  better  than  nothing  at  all,  snapshots  are  not  the  ultimate  answer  to  improving 
maintained  SA  on  operational  anomalies.  For  one  thing,  many  of  the  snapshot  facilities  cited  have  to  be 
actively  triggered  by  the  pilot.  This  means  that  correlation  of  the  snapshot’s  timeframe  with  the  timeframe 
of  the  anomalous  behavior  is  a  function  of  the  pilot’s  reaction  time.  It  is  also  reasonable  to  suggest  that  in 
the  case  of  severe  anomalous  behavior  (e.g.,  uncontrolled  movement)  triggering  a  data  snapshot  may  not 
be  the  pilot’s  most  pressing  concern. 

The  ultimate  solution  is  of  course  to  dynamically  record  as  full  a  record  of  in-flight  data  as  the  available 
hardware  permits,  so  as  to  offer  maintained  the  most  comprehensive  possible  picture  of  what  happened 
and  when  it  occurred.  This  is  the  happy  prospect  the  emerging  RQ-1  and  F/A-22  maintained  face. 
However,  it  is  currently  out  of  reach  for  maintained  working  with  the  older  aircraft  (A- 10,  F-15,  and  F- 
16).  Though  not  cheap  in  absolute  terms,  it  would  seem  proportionally  cost-effective  to  consider  adding  a 
more  comprehensive  onboard  data  capture  and  recording  capability  to  these  older  aircraft.  More 
expensive  still  would  be  the  prospect  of  adding  in-flight  data  capture  combined  with  real-time  telemetry. 
This  more  complicated  approach  would  offer  the  prospect  of  giving  more  maintained  the  clear 
advantages  enjoyed  by  the  RQ-1  personnel. 

4.3.5  The  Advantage  of  Better  Up-front  SA  versus  Probabilistic 
Prediction  of  Likely  Faults 

Earlier,  in  discussing  the  CTA,  it  was  claimed  that  the  best  way  to  get  a  good  “head  start”  on  traversing 
even  a  truncated  “quick  fix”  process  path  is  to  obtain  a  decent  picture  of  what  went  wrong  in  this 
particular  case.  The  qualification  with  respect  to  “particular  case”  is  very  relevant  to  evaluating 
informational  interventions,  because  it  illustrates  the  higher  relative  criticality  of  initial  situation 
awareness  in  the  particular  case  vedus  situation  awareness  of  the  general  class  of  cases.  Flight  control 
maintenance  is  done  on  a  recurring  basis.  This  means  that  in  the  aggregate  one  can  evaluate  the  relative 
incidence  of  candidate  fault  conditions  and  provide  a  statistical  basis  for  “playing  the  odds”  when  initially 
hypothesizing  what  fault  may  underlie  the  current  case.  However,  it  is  just  as  true  that  each  time  a  flight 
control  problem  is  reported,  it  is  reported  for  this  aircraft  and  in  the  context  of  these  in-flight  conditions 
and  behaviors.  In  other  words,  maintained  don’t  deal  with  the  aggregate;  they  deal  with  the  particular. 

This  limits,  but  does  not  negate,  the  potential  applicability  of  statistical  aids  in  predicting  diagnoses  for 
one  or  another  specific  case.  Still,  this  limitation  is  such  that  it  is  safe  to  claim  that  good  particular 
information  up  front  (e.g.,  at  the  Problem  Reporting  step)  is  more  likely  to  improve  overall  process  path 
performance  than  advice  on  the  odds  of  it  turning  out  to  be  this  or  that  based  on  prior  cases.  In  any  case, 
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the  surprisingly  substantial  proportion  of  maintenance  “quick  fixes”  purported  to  go  unreported 
diminishes  the  confidence  one  could  attribute  to  statistics  derived  from  past  documentation. 

4.3.6  The  Prospects  for  Better  Capture  and  Dissemination  of 
Experiential  Knowledge 

As  has  been  discussed  multiple  times  in  the  course  of  this  report,  the  SMEs  repeatedly  cited  the 
importance  of  experiential  knowledge.  All  types  of  SMEs  interviewed  (e.g.,  maintainers,  riggers, 
managers,  and  trainers)  indicated  experiential  knowledge  (tips,  hints,  lore)  was  an  effective  facilitating 
factor  in  improving  maintenance  task  performance.  The  SMEs  just  as  uniformly  noted  that  there  are  few 
if  any  mechanisms  in  place  for  capturing  such  knowledge  and  even  fewer  channels  for  disseminating  it. 


The  SMEs  consistently  stated  the  logbooks  are  a  very  useful,  very  important,  and  often  overlooked  source 
of  information  and  experiential  knowledge.  They  were  unable  to  cite  any  other  significant  means  for 
capturing  experiential  knowledge  in  place  at  this  time.  However,  this  deficiency  is  apparently  going  to 
improve  with  the  arrival  of  more  comprehensive  and  integrated  data  resources  associated  with  the  latest 
aircraft.  In  particular,  the  F/A-22  IMIS-based  system  permits  the  addition  of  history  notes  into  the 
database.  The  SMEs  often  cited  this  oncoming  capability  and  suggested  it  would  be  useful  to  have 
available  for  all  aircraft  types. 

There  are  two  straightforward  innovation  paths  that  would  reasonably  address  this  deficiency: 

♦  Improve  capture  and  collation  of  logbook  entries.  This  is  easier  said  than  done  -  especially  if  this 
were  to  be  pursued  in  the  most  obvious  fashion  (digitization).  For  one  thing,  there’s  an  alleged 
correlation  between  maintainer  experience  and  general  computer  aversion.  This  suggests  a 
computerized  logbook  capability  might  not  be  used  (or  not  be  as  well  used  as  hoped)  by  the  senior  or 
experienced  maintainers  presumably  serving  as  the  main  sources  for  such  knowledge.  Another  issue 
concerns  privacy  and  confidentiality.  Maintainers  are  unlikely  to  entrust  lessons  learned  to  a  digital 
medium  if  such  reports  reflect  badly  on  them  personally  (e.g.,  warnings  about  one’s  mistakes  made) 
or  provide  evidence  of  unsanctioned  procedures  (e.g.,  a  trick  of  the  trade  whose  execution  requires 
violating  standard  practices  or  rules). 

♦  Provide  effective  channels  for  disseminating  access  to  capture  experiential  knowledge.  There  are  a 
number  of  issues  to  be  decided  in  trying  to  make  experiential  knowledge  available  to  a  wider 
audience.  Should  it  be  offered  in  a  “supply-push”  manner  (e.g.,  a  print  publication  or  email 
newsletter)  or  on  a  “demand-pull”  basis  (e.g.,  a  central  data  library  or  bulletin  board)?  In  the  long  run, 
the  approach  most  likely  to  be  both  effective  and  efficient  would  be  a  ListServ-style  discussion  forum 
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combining  login  access  to  a  data  repository  along  with  optional  subscriptions  to  topically-delineated 
message  threads. 

4.3.7  The  Prospects  for  Addressing  Novice  Deficiencies  in 
Background  Knowledge 

The  KA  indicates  reasonable  consensus  that  novice  or  younger  maintained  are  deficient  in  general 
technical  knowledge,  appreciation  for  operational  context  influences  on  actual  maintenance  work,  and 
understanding  of  the  inner  workings  of  the  aircraft  and  aircraft  subsystems  they’re  expected  to  service. 
Strictly  speaking,  the  means  for  addressing  these  deficiencies  fall  largely  outside  this  project’s  purview. 
Because  the  general  and  aircraft-specific  technical  knowledge  deficiencies  are  clearly  subjects  to  be 
addressed  with  respect  to  training,  they  lie  outside  the  scope  of  the  maintenance  process  itself.  A  greater 
appreciation  for  the  exigencies  and  influences  of  real-world  situational  context  is  somewhat  more 
addressable  in  conjunction  with  daily  maintenance  work.  However,  this  issue  still  would  seem  to 
insinuate  innovation  in  terms  of,  for  example,  personal  mentoring  with  more  experienced  personnel,  and 
not  an  “information”  or  “technical”  innovation  of  the  sort  the  team  set  out  to  examine.  By  the  same  token, 
any  improvements  in  capturing  and  making  available  the  experiential  knowledge  that  has  been  repeatedly 
emphasized  would  peripherally  aid  novices  in  learning  how  maintenance  work  is  actually  performed. 
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5  Technology  Search 

5.1  The  Decision  Criteria  Development  Workshop 

To  provide  high-level  system  requirements  needed  to  facilitate  a  broad  technology  search,  the  MXM 
Team  hosted  a  Decision  Criteria  Development  Workshop  30-31  July  2003  at  the  Northrop  Grumman 
facility  in  Fairborn,  Ohio.  Originally  scheduled  for  May,  it  was  slipped  as  a  result  of  KA  trips  being 
pushed  back  due  to  base  access  problems  discussed  earlier  and  because  of  on-going  heavy  taskings  which 
made  it  difficult  for  potential  attendees  to  commit  to  coming  to  the  workshop. 

The  purpose  was  to  bring  together  experienced  field  level  maintainers  and  their  leaders  in  a  structured 
environment  to  gain  their  input  on  what  a  potential  tool  or  set  of  tools  to  enhance  diagnostics  and 
troubleshooting  should  generally  do. 

The  workshop  began  with  the  team  providing  background  and  an  introductory  MXM  project  overview  to 
orient  everyone  to  the  goal  at  hand.  That  done,  the  lead  facilitator  gave  a  brief  description  of  the  process 
that  would  be  followed  and  the  tools  that  would  be  used  to  capture  and  manipulate  the  data.  After  a  short 
question  and  answer  session  the  group  began  the  process  of  developing  the  decision  models  upon  which 
everything  else  would  build. 

5.1.1  Structured  Decision  Making  Process 

There  are  several  benefits  to  the  structured  decision-making  process  that  the  team  employed  during  the 
workshop.  The  group  is  lead  through  a  comprehensive  process  that  encompasses  all  the  inputs  of  the 
attendees.  This  allows  file  group  to  focus  on  developing  decision  models,  while  the  facilitating  team 
focuses  on  the  process.  In  addition,  because  a  diverse  group  is  involved  in  the  development  of  the 
decision  model,  there  is  an  increased  opportunity  to  gather  wider  ranging  ideas.  Group  consensus  is  also 
reached  as  the  group  has  an  opportunity  to  compare  and  contrast  the  various  criteria  against  each  other. 
Finally,  the  use  of  a  structured  process  enables  clear  documentation  showing  how  the  decision  model  was 
developed.  In  guiding  the  group  through  the  process  a  funnel  approach  was  used  to  move  them  from  very 
broad  concepts  and  gradually  bring  them  to  a  more  specific  focus.  This  approach  is  depicted  in  Figure  4. 
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Figure  4:  The  Funnel  Approach 

5.1. 1.1  The  Decision  Models 

In  order  to  set  the  boundaries  for  the  decision  models,  the  facilitator  asked  the  group  the  following 
question:  “What  can  be  done  to  improve  the  aircraft  maintenance  technician’s  performance  by  improving 
their  troubleshooting  capability  and  identifying  the  information  required  for  technicians  to  better  schedule 
aircraft  repairs  based  on  imminent  system  failures?” 

The  group  then  generated  several  ideas  through  a  “brainstorming”  session.  The  facilitator  then  grouped 
similar  ideas  together  and  the  grouping  was  reviewed,  discussed  in  detail,  and  validated  by  the  attendees. 
This  resulted  in  seven  possible  decision  models: 

♦  Deployability 

♦  Tech  Data 

♦  Hardware 

♦  Prognostics/Scheduling 

♦  Training 
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♦  Corporate  knowledge 

♦  A  new  software  development  that  contains  the  software  modules  identified  above 
The  groupings  are  listed  in  the  following  (Tables  18-24): 


Table  18:  Deployability 


Deployability 

Software  should  be  usable  on  desktop,  portable  laptop,  palm  pilot,  etc. 

♦  Portability:  End  product  needs  to  deploy  with  one  airframe/person 

4  ^kS^CDSvD)ed  t0  tr°UbleSh00t  (G0'81 1  CAMS)-  Solution  needs  t0  9°  with  tro°PS  in  their 

♦  The  F16  community  is  doing  this  now  (system  in  a  pocket) 

New  diagnostic  aid  should  be  deployable 

♦  Wireless  LAN  on  flight  line  with  CAMS,  CFRS,  CFI 

♦  Able  to  troubleshoot  and  order  parts  without  leaving  the  flight  line 

Table  19:  Prognostics/Scheduling 

Prognostics/Scheduling 

Overall  system  should  get  smarter,  provide  probability  of  success  based  on  TO  recommended 
solutions 

Capture  LRU  maintenance  data.  Details  for  predictive  behavior 

♦  Failure  prediction:  if  something  was  predicted  to  fail  and  there  was  a  malfunction  in  that 
system  alert  the  end  user  to  check  that  part  first. 

♦  If  something  has  80MTBF  and  it  is  at  80  hours  of  operation,  check  this  item 

Maintenance  scheduling  tool  with  predictive  capability.  If  a  jet  is  going  into  heavy  maintenance 
tne  system  should  prompt  you  to  look  at  upcoming  items  that  are  coming  due. 

Verify  condition  of  LRUs  installed  before  removed  or/and  installed 

♦  System  troubleshooting  aid.  Gets  smarter  as  time  moves  “core"  trouble  sources 

♦  Evolve  system  to  automatically  update  FIs  or  maintenance  data  is  collected  (somethinq  like  8 

quarters  of  data  of  charts).  a 
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Table  20:  Hardware 


'"-.t':-:  ■’■'"^Hardware f 

Hands-free  interactive  TO  so  maintenance  technicians  can  inspect  more  efficiently 

Universal  test  equipment  (standardized  software) 

Diagnostic  tester  using  a  laptop  that’s  programmable  and  upgradeable  quickly 

Software  that  can  be  connected  to  a  printing  device  that  given  a  certain  task,  will  generate  all 
necessary  warning  tags 

Table  21:  Training 

Training 

Seamless  training  from  tech  schools  to  MDS.  Students  are  assigned  to: 

♦  Same  TO  formats 

♦  Intuitive  reference  system 

Write  the  training  as  an  XBox  game 

Software  training  on  in-depth  wiring  troubleshooting  this  is  becoming  a  lost  art 

Software  training  in  basic  analytical  thinking 

Table  22:  Corporate  Knowledge 

Corporate  Knowledge 

Capture  LRU  maintenance  data.  Details  for  predictive  behavior 

Capture  corporate  knowledge 

Corporate  knowledge— must  be  able  to  capture  it 

Automatic  cross  tell/sharing  of  secrets  to  troubleshooting 

If  a  Boeing  request  for  engineering  disposition  has  been  answered  previously  for  a  current  write¬ 
up  what  was  the  response? 

Communication  between  units  throughout  the  world.  There  are  problems  encountered  some 
places  that  would  be  great  for  the  rest  of  that  airframe’s  people  everywhere  would  like  to  know 

How  they  fixed  them  and  what  to  look  for. 
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Corporate  Knowledge 


Better  access  to  historical  data  on  aircraft 

Data  capture  (troubleshooting).  We  need  the  ability  to  capture  logbook  entries.  Logbooks  are  a 
hundred  times  more  descriptive  than  current  CAMS  entries.  Analysis  of  how  people  actually 
troubleshoot  the  system  and  its  fix  could  be  beneficial  to  incorporate  into  tech  data. 

Develop  a  web  portal  that  1  could  post  questions  and  answers  to.  Needs  to  include  tech  reps 
(Boeing,  Northrop,  etc.) 

When  put  something  into  CAMS/GO-81,  have  system  automatically  go  out  and  tell  you  what 
similar  gigs  have  occurred  and  what  corrective  action  were 

Capture  data  to  CAMS/GO-81  automatically.  Be  able  to  pull  all  info  into  log  books  and  have  log 
book  automatically  update  CAMS/GO-81 

Capability  to  instantly  access  aircraft  history  worldwide 

Ability  to  view  previous  solutions  to  similar  problems 

MDS  specific  automatic  cross  talk  calculated  from  maintenance  data  analysis 

TO  Aide.  Sometimes  TOs  don’t  have  all  the  information  we  need.  Fault  trees  often  lead  to  dead 
ends.  Needs  to  be  a  process  for  past  experiences  from  other  troubleshooters  who  have  similar 
problems  and  be  able  to  access  their  knowledge. 

Access/store  LRU  histories 

Access  to  AMU  log  books,  loaded  on  a  computer  for  instant  search  and  analyze  capability 

When  new  items  are  developed  and  purchased  for  airframes  get  all  information  associated  with 
those  items  form  the  companies  and  engineers.  To  often  companies  out  there  hold  certain 
information  back  from  the  people  working  the  jets  in  hopes  they  can  get  more  money  (i.e.,  BIT 
codes  that  are  not  diagnosable  or  understandable  by  the  maintainers). 

♦  Ability  to  strip  CAMS/G081  of  all  relevant  data. 

♦  We  should  define  data  sets 

♦  Technician  needs  complete  access  to  post  maintenance  actions  on  jet  and  LRU  history 

System  status  and  BIT  to  maintainers  before  aircraft  lands 

Instantly  usable  aircraft  data.  Not  have  to  go  to  manufacturer  to  analyze  download 

Elimination  of  false  BIT  codes 

Capture  aircraft  flight  data 

Translation  of  BIT  data  to  usable  information 

Eliminate  ambiguous  BIT  solutions 
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Corporate  Knowledge 

Accurate  recording  of  flight  parameters  at  time  of  fault 

With  aircraft  in  chocks  ability  to  interface  with  a  running  aircraft  (read  MFLs,  view  switches,  etc.) 

Table  23:  Tech  Data 

;>V  Tech  Data 

Overall  system  should  get  smarter,  provide  probability  of  success  based  on  TO  recommended 
solutions 

Software  that  will  allow  C-17  technician  to  order  correct  part/software  #  for  specific  tail  number 

Automated  update  of  time  changes  when  an  item  is  changed  for  unscheduled  maintenance 

Interactive  wiring  diagrams  that  are  aircraft  specific.  Be  able  to  plug  a  PDA  into  job  and  it 
automatically  “knows  the  job"  with  all  flags  and  updates 

TOs/Display  system  for  easier  reading  of  schematics 

Laptop  system,  windows  based,  that  incorporates  FIs,  TOs,  Gee  Wiz  information,  etc. 

Electronic  tech  data.  JG,  WD,  SD,  FI  on  flight  line  using  hyperlink 

♦  Tech  data  (general  00-series)-  general  troubleshooting  procedures  non-system  specific. 

♦  Someplace  to  start  when  no  aircraft  TO  is  available 

Tech  Order  “movies"  that  are  interactive 

Availability  of  digital  tech  data 

♦  Software  that  will  tell  technician  instantly  what  tools  (i.e.  socket,  wrench)  will  be  needed  to  do 
a  certain  task. 

♦  Tells  maintainer  which  toolbox  to  take  to  the  line. 

Hands-free  interactive  TO  so  maintenance  technicians  can  inspect  more  efficiently 

Data  capture  (troubleshooting).  We  need  the  ability  to  capture  logbook  entries.  Logbooks  are  a 
hundred  times  more  descriptive  than  current  CAMS  entries.  Analysis  of  how  people  actually 
troubleshoot  the  system  and  its  fix  could  be  beneficial  to  incorporate  into  tech  data. 
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Table  24:  Not  Assigned  to  a  Group 


Not  Assigned  to  a  Group 

Self-healing  systems  (magic  fairy  dust) 

Automatic  location  tracking  of  age  and  CTK  items  (RFID?) 

Reduce  number  of  wires  on  aircraft  (data  bus?) 

Deviation  from  organization  structure  to  maximize  the  skilled  people 
Scheduling  system  “more  efficient  use  of  available  skills” 


Improved  personal  transport  for  flight  line  personnel-each  tech  has  mobility 


From  this  list  the  group  was  asked  to  prioritize  the  groupings,  now  called  modules,  by  casting  votes  for 
the  ones  they  considered  most  important.  Corporate  Knowledge  was  identified  as  the  most  import, 
followed  by  Tech  Data  and  Training.  It  should  be  noted  that  this  matches  quite  closely  with  the  emphasis 
areas  that  were  seen  during  the  KA  trips. 

5.1.1 .1.1  Developing  the  Decision  Models 

The  facilitator  then  used  the  Expert  Choice  tool  to  develop  and  weight  each  decision  model.  Expert 
Choice  uses  the  Analytical  Hierarchy  Process  (AHP)  to  develop  decision  models.  The  AHP  is  a  powerful 
and  comprehensive  methodology  that  uses  a  hierarchical  model  comprised  of  a  goal,  criteria,  and  sub- 
criteria  for  each  decision.  This  hierarchical  approach  is  very  common  when  making  decisions  with 
multiple  objectives.  Using  the  AHP  enables  the  decision-maker  to  derive  weights  as  opposed  to  arbitrarily 
assigning  them.  The  AHP  also  allows  decision-makers  the  capability  to  incorporate  both  objective  and 
subjective  considerations  in  the  decision-making  process.  The  AHP’s  flexible  and  efficient  hierarchical 
framework  guides  a  decision  group  to  an  agreed  upon  conclusion.  Because  all  parts  of  the  hierarchy  are 
interrelated,  it  is  easy  to  see  how  a  change  in  one  factor  will  affect  all  the  other  factors. 

5.1. 1.1. 1.1  Decision  Model  Goals 

For  each  decision  model,  the  facilitator  asked  the  group  develop  a  goal  for  the  decision  model.  Those 
goals  are  depicted  in  Table  25. 
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Table  25:  Decision  Model  Goals 


Model 

Goal 

Corporate  Knowledge 

Allow  technicians  to  have  ready  access  to  information  about  the 
system 

Tech  Data 

Instant  access  to  accurate  approved  guidance  for  system  repair 

Training 

Technicians  are  trained  to  maximize  their  trouble  shooting  skills 

New  Software  System 

Improve  field  level  troubleshooting  capability  worldwide 

Hardware 

Enable  worldwide  wireless  connectivity 

5.1. 1.1. 1.2  Criteria 

Once  a  goal  was  established  for  each  model,  the  group  was  asked  to  identify  criteria  that  would  be  used  to 
make  sure  the  goal  was  met.  The  facilitator  used  a  Round-Robin  technique  to  gather  die  criteria  from  each 
attendee.  All  the  criteria  were  then  captured  in  Expert  Choice’s  Structuring  Mode,  with  no  discussion 
about  any  of  them.  Once  all  ideas  were  captured,  the  group  refined  the  criteria  by  combining  similar  ideas 
and  clarifying  the  criteria  that  were  presented.  This  is  where  the  bulk  of  the  group  discussions  took  place, 
as  individuals  further  clarified  their  criteria.  In  preparation  for  assigning  weights  to  the  criteria,  the 
individual  criteria  were  grouped  into  common  areas,  thereby  developing  criteria  and  sub-criteria.  Figure  5 
shows  the  criteria  and  groupings  for  the  Tech  Data  decision  model. 
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Figure  6  shows  the  criteria  and  groupings  for  the  Training  decision  model. 


Figure  6:  Training 
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Figure  7  shows  the  criteria  and  groupings  for  the  Corporate  Knowledge  decision  model. 


Figure  7:  Corporate  Knowledge 


Once  the  group  agreed  on  the  appropriate  groupings  for  the  criteria  and  sub-criteria,  the  facilitator  used 
Expert  Choice’s  Evaluation  &  Choice  mode  to  establish  weights  for  each  standard.  The  group  made  pair¬ 
wise  comparisons  of  the  criteria  within  each  grouping.  For  each  criterion,  the  group  was  asked  which  is 
more  important:  “a”  or  “b,”  “b”  or  “c”  and  “a”  or  “c,”  etc.  The  number  of  pair-wise  comparisons  required 
for  each  level  of  the  decision  model  is  determined  by  the  following  formula:  (n)(n-l)/2.  For  three  criteria, 
there  would  be  (3X2)/2  =  3  judgments,  while  four  criteria  would  require  (4)(3)/2  =  6  judgments.  The 
□umber  of  judgments  is  represented  by  colored  squares  in  the  lower  right  comer  of  the  screen.  Figure  8 
shows  the  process  comparing  Data  and  Platform  for  the  Tech  Data  decision  model. 
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Figure  8:  Pair-wise  Comparison 

The  group  judged  Data  (how  data  is  stored  and  maintained)  to  be  more  important  than  Platform  (hardware 
and  operating  systems).  Once  comparisons  were  made  between  all  criteria,  the  results  are  displayed  in 
Figure  9. 
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Based  on  the  judgment  of  the  group  when  looking  at  two  items  at  a  time,  the  most  important  criterion  is 
Data,  followed  by  User  Interface,  System  Integration,  and  finally  Platform.  The  top-level  comparisons  for 
Tech  Data  also  had  an  inconsistency  ratio  of  0.01 .  The  inconsistency  ratio  shows  how  well  the  group 
compared  the  items.  An  inconsistency  ratio  shows  the  percentage  of  time  the  group  was  inconsistent  in 
making  judgments  on  a  set  of  particular  elements.  An  inconsistency  ratio  of  0.00  indicates  the  group  was 
always  consistent  when  making  judgments,  while  an  inconsistency  ratio  of  1.0  indicates  the  group  was 
always  inconsistent  when  making  judgments.  In  general,  any  inconsistency  ratio  less  than  0.1  is 
acceptable. 


Table  26  shows  the  overall  inconsistency  ratio  for  each  model  the  group  developed. 


Table  26:  Inconsistency  Ratios 


Model 

Inconsistency  Ratio 

Corporate  Knowledge 

0.05 

Tech  Data 

0.02 

Training 

0.02 

New  Software  System 

0.0 

Hardware 

0.05 

5.1 .1.1 .2  Standards 

After  the  criteria  for  the  decision  models  were  identified,  grouped,  and  weighted,  the  group  began  the 
process  of  identifying  standards  for  each  of  the  criteria.  Standards  are  used  to  judge  how  well  an 
alternative  (in  this  case,  software  package)  meets  the  criteria.  Most  MXM  criteria  are  judged  using  three 
categories  (high,  medium  and  none)  for  each.  Other  MXM  criteria  are  judged  using  only  two  categories 
(High  or  none).  Standards  are  given  a  numerical  rating  using  a  logarithmic  scale  so  that  alternatives  that 
provide  a  great  value  to  the  organization  come  out  at  the  top  of  the  list,  while  those  alternatives  that  do 
not  provide  a  benefit  to  the  organization  drop  to  the  bottom  of  the  list.  The  standard  scale  for  the  MXM 
analysis  is: 

♦  High- 100% 

♦  Medium-  33% 

♦  Low-0% 

A  total  benefit  score  for  an  alternative  is  derived  using  the  following  formula: 

Total  Benefit  Score  =Criteria  1  Weight  *  Standard  1  Weight  + 

Criteria  2  Weight  *  Standard  2  Weight  + 

...+ 
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Table  27  shows  the  standards  for  the  user  interface  grouping  of  the  Tech  Data  model. 


Table  27:  Standards  for  Tech  Data  User  Interface 


Criteria 

High 

Medium 

None 

Multiple  media  (video,  audio,  text,  picture).  The  group 
would  like  to  have  the  data  displayed  in  a  variety  of 
formats. 

All 

One  or 
more 

None 

Highlighted/zoomable  all  data.  The  group  would  like  to 
be  able  to  highlight  portions  of  data  and  zoom  into 
parts  of  a  diagram.  This  is  especially  important  for 
wiring  diagrams.  They  mentioned  a  specific  tool  called 
Wire  Illuminator  that  has  this  capability 

Yes 

No 

Split-screen  capability.  This  would  allow  the  group  to 
view  a  drawing/video  while  at  the  same  time  reviewing 
a  textual  description. 

Yes 

No 

Easy  navigation  for  the  end  user-similar  to  internet 
explorer 

Internet 

Explorer 

Familiarity 

Intermediate 

I 

Required  equipment/materials  list.  A  listing  of  tools, 
parts,  etc.  that  are  required  for  the  repair.  This  enables 
the  technician  to  gather  all  required  equipment  and 
materials  prior  to  starting  the  maintenance  action  and 
shorten  the  maintenance  time. 

Yes 

No 

Ability  to  add  personal  notes  linked  to  logbook.  This 
allows  the  maintainer  to  keep  personal  notes  that  aren’t 
shared  with  the  global  community. 

Yes 

No 

The  complete  listing  of  standards  for  all  criteria  can  be  found  in  the  section  for  each  decision  model. 
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5.1 .1 .1 .3  Corporate  Knowledge  Decision  Model 

During  the  initial  discussions  the  team  recognized  Corporate  Knowledge  as  the  area  that  could  provide  the  greatest  impact  in  improving 
maintenance  troubleshooting  times.  The  decision  model  (Figure  10)  shows  the  structure  the  team  developed  to  prioritize  requirements  for  a 
corporate  knowledge  software  system. 
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Corporate  Knowledge 

Allow  technicians  to  have  ready  access  to  information  about  the  system 
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Figure  10:  Corporate  Knowledge  decision  model 
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log  book).  The  group  would  like  the  ability  to  Access  some 

Interface  interface  directly  with  other  Air  Force  systems.  0.039  Access  all  systems  systems  No  access 


Capture  log-books-selective  capture  of  logbook 
information.  The  team  would  like  to  be  able  to 
identify  the  text  that  is  shared  with  the  world  versus 

Data  store  the  text  that  is  kept  at  the  base/repair  location.  0.002  Selective  capture  Capture  all  Capture  none 
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provides  a  graphical  representation  of  the  weights  of  die  group  assigned  to  each  of  the  requirements  for  Corporate  Knowledge, 
fit  within  the  graph,  the  descriptions  of  the  criteria  have  been  shortened. 


Figure  11:  Prioritization  of  Corporate  Knowledge  Requirements 


Ready  access  to  tech  data  and  wiring  diagrams  was  another  key  factor  in  improving  maintenance  troubleshooting  times.  The  group  developed  the 
following  decision  model  (Figure  12)  to  capture  the  criteria  that  are  important  for  a  tech  data  software  system. 
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Access  to  Base  MOIs  and  locdt  regulations. 
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Data  Keyword  search  to  a  list  of  related  TOs  (Google)  b.056 
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Flag  and  report  dead-ends/discrepancies  in  Tech  data.  This  is  meant 
to  help  improve  the  tech  data  by  identifying  paths  that  do  not  resolve 
problems.  The  group  mentioned  several  cases  where  Tech  Order  1 

referred  to  Tech  Order  2,  which  referred  to  Tech  Order  3,  and  finally  Automatic  Manual 


i  provides  graphical  representations  of  the  weights  the  group  assigned  to  each  of  the  requirements  for  Tech  Data, 
fit  within  the  graph,  the  descriptions  of  the  criteria  have  been  shortened. 


Figure  13:  Prioritization  of  Tech  Data  Requirements 


IO 


g> 


I 


&  i 


at  *3 


do 


C0 

■1 

•3 

£ 

4) 

i 

o 

% 

fc 

*5 

{2 

cd 

o 

a 

Cfl 

126 


Training 

Technicians  are  trained  to  maximize  their  Troubleshooting  skills 
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Imaintainer  to  understand  what  happens  if  they  fail  to  comply 
kith  maintenance  standards. 
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BEST  AVAILABLE  COPY 


Figure  15:  New  Software  System  Decision  Model 
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Able  to  access  real-time  information  at  the  jet/LAN-independent.  The  team  wants  to  have  ready  access  0.01 821 3 
corporate  Knowledge  to  information  about  a  specific  tail  without  being  connected  to  a  LAN. 


Keyword  search  to  a  list  of  related  TOs  (Google)  b.016041 
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Pass/fail  ladder-structured  progression.  This  will  allow  the  maintainer  to  re-take  courses  hen  there  is  a 
raining  need  to  refresh  skills  for  an  activity  that  has  not  been  performed  for  a  period  of  time. 
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lility  to  contact  Subject  Matter  experts-system  can  contain  email  links  or  other  contact  information  b.004933 
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Corporate  Knowledge  knowledge  and  other  data  systems. 
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identifies  the  Training  Requirements. 


New  Software  System-1 


CK-Ease  of  usage  for  indMduaJf 
(^-Prevention  violation  of  safety  standards  7D  and  regulator* 
Tr-PrtMHTtVfo«ta 
Tr-DetBuftfT»dutefofbask:trouW 

Tr-Scenario  Bated  Training. 
Tr-£as*y  updateatomaintain  current  information. 
TD-Rter  TD  by  MDS/tafl  number  (also  systemfeupport  equip) 
Tr-Searchabte  Training 
Tr^Analyfcai  thinking  Training  guides 
Tr-User  feedback  in  sknutatfons. 
TD-Easy  navigation  for  the  end  user-eknlar  to  intomet  explorer 
CK-FHter  based  on  criteria  entered  by  user-good  search  engine 
Tr-TraWng  access  by  capabHty  (beginner,  intermediate,  advanced). 

TD-4tighlghtBd/zoomabteald^ 
TO-Warteride  access  to  current  TD  (support,  general,  weapon  system,  tafl). 
CK-Auto  htetory/prognostics  on  •  job^earch  by  parttai  number  worldwide 
TD-Bectronic  format  with  accurate  hyperfnks. 

Tr-Basic  Wire  maintenance. 
TD-Auto  update  by  MDS/taB  number  (TCTCVmod  comptorce) . 
CK-^ble  to  access  real  time  information  at  the  jotflarvfcdependent 
TD-Key  word  search  to  a  1st  of  rented  TOs  (google) 
TD-Jnteraction  with  other  data  systems  (CAMS,  G081 ,  CFRS.  etc.) 

TD-Mu«ipte  media  (video,  audio,  text  picture). 
CK-System  captures  Cross  tel  notes  and  lessons  teamed 
CK-Systems  interface  (CAMS,  CFRS,  G081,  electronic  log  book). 

TD-Sp#  screen  capaMfy. 
TD-Smal  footprint  (portable) 
TD-Hardware  independent  (COTS  system) 
TD-Consistent  format  for TO  (fisptay 
Tr-Foime  &  supply  doc  Trtenfog 

Tr-Tefc  the  maintafoer  why  something  is  being  done  the  way  it  is  being  done. 

Tr-Provide  alternatives  to  preclude  dead  ends 
Tr-Patt/fol  ladder-stnictured  progression. 
Tr-Customfeabte  Training  paMexbto. 


Figure  16:  New  Software  System,  Part  1 
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New  Software  System-2  • 

0  0.02  0.04  0.06  0.08  0.1  0.12  0.14  0.16  0.18  0.2 

TD-Required  equipment/materials  list 
Tr-Compare  and  contrast  similar  systems  (radar,  etc). 

TD-Cautions/wammgs  are  displayed  and  highlighted. 

TD-Access  to  Base  MOIs  and  local  regulations. 

TD-Print  prefilled  warning  tags  from  selectable  list  (DD1492,  etc.) . 

TD-Flag  and  report  dead-ends/descrepencies  in  TD. 

TD-Abifityto  add  personal  notes  linked  to  logbook. 

CK-Controlled  access  based  on  user  profile 
Tr-Consequences  of  failure  for  non-compliance. 

TD-Print  capability 

CK-AbilHy  to  contact  Subject  Matter  experts-email/olher  contact  info 
CK-AbBity  to  download  to  CD/DVD/print 
TD-Unk  to  AFTO  22/135  to  provide  updates 
CK-Data  maintenance-ability  to  add/delete  as  data  moves  to  TO. 

CK-Aircraft  Interface.  Ability  to  interface  directly  with  the  aircraft 
Tr-Task  completion  time  standards. 

CK-Video  and  voice  capability 
CK-Chat  room  capabffity-wlthin  malntainers. 

Tr-Provkie  sources  for  Training  devices  for  hands-on  Training. 

CK-TD  history-has  it  been  addressed  if  not  in  TO. 

CK-Acoept  digital  pictures 
CK-Comptete  and  accurate  data. 

CK-Part/equipment  availability. 

CK-Easy  to  update  date  in  both  CK  and  other  data  systems. 

CK-Gathered  data  should  be  driven  to  TD  /FIT. 

CK-Capiure  log-books-selective  capture  of  logbook  information. 

Tr-Proactive  Tr.  This  allows  a  technician  to  take  Tr  before  it  is  required. 

CK-Expandable.  Easily  expandable  as  new  components  are  identified. 

CK-Prevent  dual  entry-this  system  and  other  systems. 

CK-Indudes  TOs  and  wiring  diagrams. 

Tr-Training  should  offer  historical  and  trivial  tips. 

Tr-A  “How-to-  link  to  the  Training  module/TD 
Tr— Tutorial  for  this  Maintenance  Mentor  System 
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Figure  17:  New  Software  System,  Part  2 
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Once  the  group  completed  identifying  and  weighing  all  the  software  requirements,  they  performed  the  same  process  for  hardware.  The  decision 
model  (Figure  18)  identifies  the  groupings  and  criteria  for  the  hardware  needed  to  support  the  software  system. 
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columns  show  the  standards  the  group  identified  for  each  criterion. 


Table  32:  Hardware  Requirements  in  Priority  Order 
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Basic  design  Modular/Upgradeable  b.025 
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BEST  AVAILABLE  COPY 


5.2  Potential  Tool/Technology  Options 
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Advertised  The  purpose  of  the  APU  Tester  is  to  provide  the  aircraft  operator  a  tool  for 

Features  troubleshooting,  testing  and  problem  isolation  of  the  Allied  Signal  GTCP-85-129CK 

Auxiliary  Power  Unit  (APU).  Use  of  this  tester  will  reduce  the  number  of  premature 
engine  removals  on  the  737-100/200/300  Boeing  Commercial  aircraft. 


Advertised  The  purpose  of  the  APU  Tester  is  to  provide  the  aircraft  operator  a  tool  for 

Features  troubleshooting,  testing  and  problem  isolation  of  the  Allied  Signal  GTCP-331  -200 

Auxiliary  Power  Unit  (APU)  and  the  Engine  Control  Unit  (ECU).  Use  of  this  tester  will 
reduce  the  number  of  premature  engine  removals  on  the  757/767  Boeing  Commercial 
aircraft. 
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Web-Based 
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151 


152 


153 


154 


Extensive  Offline  Diagnostics 
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Provides  Information  for  Deciding  How  and  When  to  Change  Operations  to  Avoid 
Failure 
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possible  within  the  framework  of  the  given  business  processes.  Issues  with  COTS  products  include  product  inflexibility,  unwanted  or  missii 
features,  system  incompatibilities,  and  integration  difficulties,  to  name  just  a  few. 
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Appendix  B:  Glossary 


ACC  -  Air  Combat  Command 

AFETS  -  Air  Force  Engineering  &  Technical  Services.  Organic  specially  trained  USF  civil  service 
technicians  who  provide  technical  and  advisory  support  to  the  maintainers. 

AFRL/HESR  -  The  Logistics  Readiness  Branch  of  the  Deployment  and  Sustainment  Division  within  the 
Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate;  sponsor  of  the  Maintenance  Mentor 
research 

AHP  -  Analytical  Hierarchy  Process 
AMC  -  Air  Mobility  Command 
AOA  -  Angle  of  attack 

AR  -  Aero-Repair  Flight  in  the  Maintenance  Squadron  (commonly  called  the  AR  Shop  or  AR,  also 
known  as  Repair  and  Reclamation  -  a  label  for  the  riggers 

BIT  -  Either  (a)  “built-in  test”  (capability  for  testing  using  onboard  integrated  equipment)  or  (b)  “binary 
digit”  (a  data  result  from  testing) 

CADC  -  Central  Air  Data  Computer.  An  onboard  computer  providing  basic  flight  data  to  various  systems 
in  the  aircraft 

CAMS  -  Core  Automated  Maintenance  System 
CCU  -  Communications  Control  Unit  (C-17) 

CSFDK  -  Crash  Survivable  Flight  Data  Recorder. 

CFRS  -  Computerized  Fault  Reporting  System  (F-15) 
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COTS  -  Commercial  Off  the  Shelf 


CSMU  -  Crash  Survivable  Memory  Unit  (F/A-22) 

DFLCS  -  Digital  Flight  Control  System 

DHM  -  Diagnostic  Health  Management.  Built-in  fault  tracing/diagnostic  capabilities  on  the  new  F/A-22 
Raptor. 

DTC  -  Data  Transfer  Cartridge  (F/A-22  in-flight  data  capture  device). 

DTM  -  Data  Transfer  Module  (F  -  1 5  non-volatile  data  capture  device). 

DTOS  -  Digital  Technical  Order  System  (computer-based  documentation  for  C-17) 

EDNA  -  Enhanced  Diagnostic  Aid 

EFIS  -  Electronic  Flight  Instrument  System  (C-17) 

flight  control  -  Non-standard  acronym  used  in  this  report  for  “flight  control.” 

FI  -  Fault  isolation  guide.  An  Air  Force  TO  used  for  maintenance  troubleshooting.  Fault  isolation  ready 
reference  cards,  although  not  official  TOs,  were  often  referred  to  as  FIs  as  well. 

FM  -  Flight  Manual 

“FITS  Tester”  -  A  label  used  to  denote  the  AN/ASM  497  Flight  Control  Test  Set  used  with  the  F-15. 
Pronounced  “flits  tester.” 

FRC  -  Fault  Report  Code  (F/A-22  formal  fault  indicators). 

FSU  -  Flight  Situation  Unit  (Predator  onboard  subsystem). 
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FTD  -  Field  Training  Detachment 


GCS  -  Ground  control  station  (Predator). 

IMIS  -  Integrated  Maintenance  Information  System 
IPB  -  Illustrated  Parts  Breakdown 

JFS  -  Jet  Fuel  Starter  used  on  the  F-15  and  F-16.  The  Falcon  (F-16)  group  mentioned  among  the 
equipment  shared  with  other  groups/units  a  laptop  for  running  “JFS”  aids  or  tool(s). 

LRU  -  Line  replaceable  unit.  An  aircraft  component  or  unit  set  of  components  capable  of  replacement  on 
the  flight  line. 

MXM  -  Maintenance  Mentor  -  (a)  The  title  of  the  project  under  which  this  report  was  done,  (b) 
Informally,  the  label  used  to  denote  a  system  or  product  deriving  from  this  effort. 

MFL  -  Maintenance  Fault  List.  A  list  of  fault  codes  or  indices  used  as  procedural  pointers  during 
maintenance.  As  evidenced  during  the  KA  sessions,  this  term  is  also  loosely  used  to  connote  any  of  the 
indices  therein. 

MX  -  Acronym  for  “maintenance.” 

MXG  -  Acronym  for  “Maintenance  Group.” 

NDM  -  Naturalistic  Decision  Making  (Klein  et  al.,  1992) 

OEF  -  Operation  Enduring  Freedom 
OIF  -  Operation  Iraqi  Freedom 

OOn  A  I  ,n«p  (also  D-O-D-A  Loop)  —  Observation.  Orientation.  Decision.  Action  Loop  (cited  by  many 
and  ascribed  to  Boyd,  1987).  Taken  to  describe  a  single  iteration  of  the  cycle  proceeding  from  data 
acquisition,  through  information  integration  and  decision  making,  to  enaction  of  a  response. 
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PDR  -  Pulse  domain  reflectometer  (tool) 


PFAD  -  Predictive  Failures  and  Advanced  Diagnostics  (an  AFRL/HESR  research  program) 

PMA  -  Portable  Maintenance  Aid.  (a)  A  label  for  any  self-contained  computerized  maintenance  analysis 
and  decision  aid.  (b)  The  laptop-style  computer  serving  as  the  primary  diagnostic  and  maintajner  interface 
to  the  F/A-22  Raptor. 

PQDR  -  Product  Quality  Deficiency  Report.  The  main  document  for  noting  and  reporting  problems  with 
parts/components. 

QA  -  Quality  Assurance 

R  &  R  -  Repair  and  Reclamation  Flight  in  the  Maintenance  Squadron,  also  known  as  Aero-Repair  or  the 
AR  Shop  -  a  label  for  the  riggers. 

Red  Ball  -  A  priority  maintenance  call  for  an  aircraft  scheduled  to  take  off  in  the  immediate  future, 
typically  the  pilot/crew  is  already  at  the  aircraft. 

situation  awareness  -  “The  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and 
space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future.”  (Endsley, 
1988,  p.  97).  Acronym  =  SA. 

SPO  -  System  Program  office 

swapology  -  A  pun  on  “switchology”  introduced  by  this  author  during  the  KA  sessions  to  denote  the 
maintenance  strategy  of  “swapping  out  parts  until  the  plane  is  fixed.”  The  maintainers  seemed  to  grasp 
my  intended  meaning  right  off  the  bat,  but  I  have  no  indication  they  use  this  term  themselves. 

switchology  -  A  casual  term  for  (e.g.)  “the  science  of  configuring  the  controls,  switches,  and  settings.” 

Used  by  maintainors  to  denote  corrective  actions  consisting  solely  of  resetting  or  reconfiguring  controls, 

switches,  etc. 
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TCTO  -Time  Compliance  Technical  Order 


TDR  -  Time  domain  reflectometer  (tool) 

TO  -  Technical  Orders)  -  die  primary  reference  manuals  associated  with  a  given  aircraft 

UAV  -  Unmanned  Aerial  Vehicle 

VIT  -  Variable  Information  Table  (RQ-1  Predator) 

WOW  -  “Weight  on  Wheels”  switch 
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Appendix  C:  Knowledge  Acquisition  Plan 

(as  distributed  prior  to  KA  visits) 

TRS  D024:  MAINTENANCE  MENTOR  (MXM) 

Knowledge  Acquisition  Plan 
Nellis  AFBKA  Visit 
April  2003 

KA  Location/Date:  Nellis  AFB  Nevada  15-17  (and/or  early  18)  April  2003 


KA  Subject  Matter:  Work  Domain:  Maintenance  (maintenance):  Squadron  maintenance  activities. 


KA  Subject  Matter:  Work  Focus:  Aircraft  F-15,  F-16,  A-10 

KA  Subject  Matter:  Work  Focus:  System(s)  Flight  controls  (flight  control)  -  broadly  defined. 

KA  Subject  Matter:  Work  Focus:  SubSystem(s)  Electrical,  electronic,  hydraulic,  and  mechanical 
subsystems  participating  in  the  control  of  flight  surfaces  and  related  components  of  the  aircraft. 

KA  Subject  Matter  Experts  (SMEs): 


Frontline  maintainers  responsible  for  diagnosing  and  repairing  flight  control  problems  with  the  target 
categories  of  aircraft. 

I.  KA  Overview 

TRS  D024  is  aimed  at  demonstrating  a  capability  for  leveraging  decision  models  and  criteria  in  RADSS 
to  determine  optimum  or  preferred  strategies  for  supporting  aircraft  maintainers.  The  focus  for  this 
demonstration  will  be  flight  control  maintenance. 

NOTE:  It  is  important  to  bear  in  mind  that  the  outcomes  of  the  KA  phase  are  to  be  fed  into  the  RADSS 
framework.  As  such,  detailed  examination  and  analysis  of  individual  and  collective  maintainors’  cognitive 
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work  performance  is  less  important  than  identifying  support  strategies,  support  needs,  support  constraints, 
criteria  for  support  assessment,  and  those  quantitative/qualitative  metrics  and  measures  relevant  to  these 
criteria. 

I.A.  The  Maintenance  Domain:  3  Tiers  of  Relevant  Knowledge/Information 

In  effect,  there  are  3  “tiers”  or  “sets”  of  knowledge  and  information  which  are  relevant  to  the  MXM 
effort.  These  are  differentiated  on  the  basis  of  topical  generality.  Figure  20  illustrates  these  tiers. 
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Figure  20:  Tiers  of  Relevant  Knowledge/lnformation 


The  reason  for  differentiating  these  three  tiers  is  to  clarify  what  our  KA  targets  are  for  the  project 
(overall)  as  well  as  what  targets  are  most  likely  to  be  addressed  in  one  or  another  of  the  planned  TRS 
D024  KA  activities.  As  Figure  1  illustrates,  the  most  general  tier  (maintenance  Operations  -  General)  is 

most  likely  to  l>e  illuminated  in  the  planner!  decision  conferences,  while  the  most  particular  tier 

(maintenance  Functions  -  Flight  Controls)  is  most  likely  to  be  illuminated  in  the  on-site  KA  exercises. 
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In  the  following  sections  the  “topology”  or  “structure”  of  the  KA  effort  will  be  introduced  and  reviewed. 
The  points  made  in  these  subsequent  sections  are  based  on  a  variety  of  factors,  including: 

♦  The  stated  goals  of  the  TRS  D024  effort  (overall) 

♦  The  more  specific  goals  of  the  TRS  D024  on-site  KA  visits 
4  The  time  and  resource  constraints  pertaining  to  the  KA  effort 

♦  The  scope  and  depth  of  cognitive  task  analysis  (CTA)  appropriate  to  the  RADSS  decision  analysis 
demonstration 

♦  The  scope  and  depth  of  CTA  that  is  feasible  given  the  resources  and  timeframe  on  this  project 

II.  A  Breakdown  of  Topics/Foci  for  the  On-Site  KA  Effort 

The  end  goal  is  to  generate  a  set  of  data  about  tools  and  instruments  employed  in  flight  control 
maintenance.  This  means  gathering  that  information  most  pertinent  to  assessing  tools  and  instruments 
(broadly  defined)  will  need  to  be  prioritized. 

To  accomplish  this,  it  will  be  necessary  to  try  and  dig  into  a  series  of  aspects  of  the  flight  control 
maintenance  functions  and  related  resources.  In  the  following  subsections  the  most  important  of  these 
general  aspects  will  be  reviewed.  The  ordering  of  the  aspect  reviews  does  not  reflect  any  rigid 
prioritization  or  ranking. 

H.A.  ASPECT  0:  Flight  Control  maintenance  Operations  (General) 

Obtain  an  overview  of  the  flight  controls  maintenance  operations  as  currently  performed  for  3  classes  of 
aircraft:  F-15,  F-16,  and  A- 10.  The  scope  of  this  aspect  is  general  -  i.e.,  the  operational  context  within 
which  flight  control  maintenance  functions  are  performed.  This  aspect  includes  overhead,  administrative, 
and  organizational  factors  comprising  that  context.  Primary  attention  should  be  paid  to  factors  including: 

♦  Administrative  context  for  the  maintenance  functions53 

♦  General  data  on  maintenance  operational  units54 

♦  General  data  on  overall  unit  maintenance  operations55 

♦  General  data  on  flight  control  maintenance  operations56 


”  SUCH  ItttlUrCb  lllUltillC.  WJUUIUUiU  auuwlutc,  sloITiug,  tvUuiivul  ivquiivuwnto,  adminbtratlvc  requtromonta$  reporting  requirements,  eto. 

54  This  means  data  on  things  like  (e.g.):  staffing;  personnel;  maintenance  team  composition;  resource  allocation,  etc. 

35  This  means  data  such  as  (e.g.):  volume  of  maintenance  “cases;”  time/resource  data  on  the  overall  maintenance  operations  flow;  QC  data  cm 
overall  unit  maintenance  operations,  etc. 

56  This  includes  things  such  as  (e.g.):  frequency  of  incidents;  duration  of  maintenance  cycles  for  flight  control  issues;  QC  data  on  flight  control 
maintenance,  etc. 
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n.B.  ASPECT  1:  Flight  Control  maintenance  Functions  (Tasks/Actions/Activities) 

Obtain  an  overview  of  the  flight  controls  maintenance  functions  as  currently  performed  for  3  classes  of 
aircraft:  F-15,  F-16,  and  A- 10.  Such  overview  info  will  need  to  be  obtained  for  both  individual 
maintainers  and  flight  control  diagnostic  teams.  Primary  attention  should  be  paid  to  factors  including: 

♦  Temporal  features  of  flight  control  maintenance  actions  and  efforts57 

♦  Personnel  features  for  flight  control  maintenance  and  repair  functions58 

♦  Indications  and  warnings  motivating  maintenance  actions  on  flight  controls59 

♦  Criteria  employed  in  evaluating  flight  control  adequacy60 

♦  Range  of  physical  activities  during  a  typical  flight  control  maintenance  activity61 

♦  Range  of  discernible  time/effort  expended  for  activity  support  (as  opposed  to  direct  maintenance 
action)62 

♦  Criteria  employed  in  determining  completion  of  maintenance  activity. 

♦  Criteria  employed  in  determining  adequacy/quality  of  maintenance  outcomes. 

♦  A  general/representative  process  map  for  the  flight  control  maintenance  activity. 

♦  Breakdown  conditions63 

♦  Demonstrable  improvements/detriments64 

n»C.  ASPECT  2:  Flight  Control  maintenance  Tools  and  Instruments 

Obtain  an  overview  of  the  flight  controls  maintenance  support  tools  (instruments,  gadgets,  manuals, 
diagnostic  aids,  etc.)  currently  employed  for  the  three  classes  of  aircraft.  Such  overview  info  will  need  to 
be  obtained  for  both  individual  maintainers  and  flight  control  diagnostic  teams.  Primary  attention  should 
be  paid  to  factors  including: 


57  Such  features  include:  frequency  of  maintenance  actions;  duration  of  maintenance  actions;  time  consumption  for  diagnostic  and  repair  aspects 
of  the  maintenance  function;  time  factors  pertaining  to  team  organization,  meeting,  consultation,  etc.;  time  required  for  testing  and  flight- 
readiness  certification;  time  dedicated  to  record  keeping  and  paperwork;  time  spent  obtaining/retuming  tools  and  instruments;  time  spent  using 
tools  and  instruments. 

”  ’nicse  include:  number  of  personnel  typically  involved;  categories  of  personnel  involved;  auxiliary  personnel  involved  (e.g.,  supply,  testing); 
individual  vs.  team  involvement;  team  structure  and  organization. 

*  These  include:  type  of  I  &  W  motivating  flight  control  diagnosis/repair;  source  of  I  &  W;  channels  and  protocols  for  I  &  W  reporting 
These  include:  metrics/measures  of  adequate  control  performance;  criteria  for  mandatory  diagnosis/maintenance;  criteria  employed  in 
evaluating  maintenance  outcomes. 

61  We  need  to  lay  out  a  representative  set  of  actions,  positions,  postures,  etc.,  that  maintainers  adopt  or  execute  during  a  typical  flight  control 
maintenance  procedure.  This  entails  factors  such  as:  on-floor  vs.  on-aircraft  positions;  use  of  assistive  devices  (ladders,  lights);  moving  around, 
etc. 

62  In  other  words,  we  need  to  take  special  note  of  time  and/or  effort  that  must  be  applied  to  “get  ready  to  do  the  next  step,”  as  contrasted  wife 
actually  doing  the  next  step.  Such  digressions  include  (e.g.):  going  to  get  a  tool/instrument;  moving  to  another  location;  contacting  someone  else- 

UuUig  paperwoik,  tfievMug  a  icfcicm*  alii,  uw.  ' 

63  These  include  any  states  or  conditions  in  which  a  procedural  constraint  or  obstacle  interferes  with  efficient  and/or  effective  execution  of  fee 
maintenance  task.  Probes  for  breakdown  conditions  include  (e.g.):  persistent  “time  sinks,”  things  feat  “make  me  want  to  pull  my  hair  out,” 
“worst-case  scenarios,”  “worst  incidents,”  etc. 

64  Improvements  include  specific  items,  events,  innovations,  etc.,  which  fee  maintainers)  cite  as  having  improved  flight  control  maintenance 
functions  in  fee  historical  past  Detriments  are  fee  opposite  -  things  feat  can  be  specified  as  having  made  things  worse. 
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♦  Identification  of  tools  and  instruments65 

♦  Categorization  of  the  tools  and  instruments  employed66 

♦  Correlations  among  personnel  and  particular  tools/instruments67 

♦  Correlation  of  tools/instruments  with  flight  control  systems/subsystems68 

♦  Correlation  of  tools/instruments  with  flight  control  maintenance  procedure/process69 

♦  Availability/accessibility  for  the  tools  /instruments70 

♦  Functional  features  for  each  of  the  tools/instruments  (relative  to  the  task)71 

♦  Interface/usability  features  for  each  of  the  tools/instruments  (relative  to  the  useifs))72 

♦  Criteria  for  selecting/employing  tools/instruments73 

♦  Breakdown  conditions74 

♦  Demonstrable  improvements/detriments75 

n.D.  ASPECT  3:  Flight  Control  Maintenance  Information  Requirements 

Obtain  an  overview  of  the  information  requirements  pertaining  to  flight  controls  maintenance  for  the  3 
classes  of  aircraft.  Such  overview  info  will  need  to  be  obtained  for  both  individual  maintainers  and  flight 
control  diagnostic  teams.  With  any  luck,  there  will  be  a  leverageable  level  of  “generality”  at  which  the 
information  requirements  are  similar  across  the  3  aircraft  types.  Primary  attention  should  be  paid  to 
factors  including: 

♦  Identification  of  specific  features,  factors,  and  factoids  critical  to  flight  control  performance 
(general)76 


65  We  need  to  inventory  the  range  of  tools  and  instruments  employed  by  the  flight  control  maintenance  personnel  (individually  and/or 
collectively)  in  the  course  of  the  flight  control  maintenance  functions.  NOTE:  For  this  purpose  informational  resources  count  as  a 
"tool/instrument.  ”  For  example:  A  reference  aid  is  a  tool  just  like  a  wrench. 

“  We’ll  need  to  lay  out  a  reasonable  taxonomy  for  the  types  of  tools  employed  (e.g.,  hand  tools  vs.  documents  vs.  electronic  modules  vs. 
computers/software). 

67  This  means  we  need  to  correlate  specific  personnel  and  specific  tools/instruments  wherever  possible.  It  will  be  important  to  both  identify  tools 
used  solely  by  one  or  another  role  and  tools  employed  by  all  roles  involved  in  the  maintenance  team. 

“  To  the  maximum  extent  possible,  we  need  to  be  able  to  correlate  tools/instruments  with  the  aircraft  components/subsystems  to  which  they  are 
applied.  This  is  necessary  to  establish  a  basis  for  correlating  tools  with  steps  or  aspects  of  the  flight  control  maintenance  function  (as  that  function 
itself  correlates  with  components/subsystems). 

49  Given  a  basic  procedural/process  schema  for  the  flight  control  maintenance  function,  we  need  to  be  able  to  correlate  tools/instruments  with 
steps  or  options  or  phases  represented  in  that  schema. 

10  We’ll  need  to  know  (for  each  listed  tool/instrument)  where  it’s  located,  how  the  maintainer  accesses  it,  how  far  he/she  goes  and  how  long 
he/she  takes  to  get/return  it,  etc. 

71  We’ll  need  to  know  how  the  tool  is  employed  in  the  course  of  flight  control  diagnosis/maintenance  activity.  Examples  include:  plugged  into  the 
aircraft  for  measurement;  applied  to  physically  modify  an  aircraft  component;  kept  at  hand  for  reference,  etc. 

72  This  covers  those  features  or  factors  pertaining  to  how  someone  uses  the  given  tool/instrument  Such  features  include  (e.g.):  size/weight; 
portability;  sensory  modalities  employed  in  interaction,  etc. 

73  This  includes  any  info  on  why  someone  decides  to  employ  a  given  tool/instrument.  In  particular,  we  need  to  pay  attention  to  any  selection 

viilMta  opplivtl  in  ooleoting  among  two  or  more  toolo/incirumentc  that  would  cuffioe  for  a  cingla  toclr 

74  These  are  conditions,  states,  or  situations  in  which  a  given  tool  becomes  problematical  or  useless  to  the  task  at  hand.  Phrased  another  way,  a 
"breakdown”  (vis  a  vis  tools/instruments)  pertains  to  that  item  not  being  as  usable  or  useful  as  one  would  hope. 

75  Specific  changes  or  innovations  that  have  improved  (or  degraded)  the  applicability/utility  of  a  given  tool/instmment. 

76  This  means  we  need  to  compile  a  set  of  key  features  and  factors  used  for  reference  in  addressing  the  flight  control  systems/subsystems  by  the 
target  SMEs. 
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♦  Identification  of  specific  features,  factors,  and  factoids  critical  to  flight  control  troubleshooting77 

♦  Identification  of  those  features,  etc.,  which  serve  as  criteria  for  assessing  flight  control 
performance/functionality78 

♦  Identification  (where  possible)  of  the  scalar  or  relative  values  used  for  such  assessments79 

♦  Correlation  of  information  elements  with  tools/instruments80 

♦  Correlation  of  information  elements  with  specific  systems/subsystems81 

♦  Correlation  of  information  elements  with  the  course  of  the  maintenance  process/procedure82 

♦  Correlation  of  information  elements  with  specific  personnel83 

HI.  KA  Strategy 

The  end  goal  is  to  generate  a  set  of  data  about  tools  and  instruments  employed  in  flight  control 
maintenance.  However,  one  cannot  reasonably  focus  on  the  tools  from  the  beginning.  Something  has  the 
status  of  “tool”  on  the  basis  of  it  being  employed  in  the  course  of  a  task  -  otherwise,  it’s  basically  just  a 
“box  on  the  shelf.”  In  other  words,  interest  here  is  only  in  what  qualifies  as  a  flight  control  maintenance 
tool,  not  any  old  thing  that  is  sitting  around  available  for  use.  This  demands  need  to  operate  with  primary 
regard  to  a  model  for  a  typical/representative  flight  control  maintenance  task,  then  correlate  data  on  tools 
with  steps,  requirements,  and  procedures  delineated  within  that  model. 

m.A.  Developing  a  Schema  for  the  Flight  Control  Maintenance  Process  (General) 

This  is  the  single  most  important  thing  we  have  to  do.  It  must  be  done  early,  and  it  must  be  done  to  a 
degree  of  specificity. /detail  that  permits  us  to  leverage  the  model  to  categorize,  sort,  and  correlate  the 
other  data. 


We  want  to  pay  special  attention  to  reference  elements  used  to  address  “what’s  wrong.”  There  is  no  necessary  correlation  between  referential 
elements  used  to  address  a  system  and  those  used  in  addressing  a  sick  system.”  While  the  former  are  obviously  important  to  outlining 
maintenance  info  needs,  the  latter  are  more  likely  to  be  critical  to  understanding  info  requirements  during  the  course  of  diagnosis  and  repair 
sctf vines. 

™  In  other  words,  out  of  those  key  features/elements  identified  in  accordance  with  the  preceding  items,  we  need  to  flag  those  feat  are  themselves 
highlighted  in  evaluating  system/subsystem  performance  (both  for  assessing  faults  and  for  evaluating  maintman^  quality). 

In  other  words,  for  each  of  fee  features/elements  employed  as  criteria,  we  need  to  probe  on  the  range  of  values  applied  in  the 
measurement/evaluation  of  that  element’s  state.  These  may  be  quantitative  or  qualitative.  NOTE:  For  qualitative  criteria,  we  shall  always  attempt 
to  identify  a  2-,  3-.  or  5-point  rating  scale.  A  2-point  scale  is  for  simply  binary  conditionals  (“on/off;”  “good/bad”),  a  3-point  is  for  a  median  or 
nominal  range  wife  critical  extrema  (“high-OK-low”),  and  a  5-point  is  a  finer-grained  elaboration  on  a  3-point.  For  fee  3-  and  5-noint  scales  fee 
center  value  will  always  be  “nominal/normal.” 

"  We’ll  need  to  specify  (wherever  possible)  which  tool(s)/instrument(s)  serve  as  the  source  for  a  given  information  element.  If  it’s  available  to 
fee  mamtainer  by  directed  unaided  inspection,  we  need  to  note  feat  as  well. 

*'  U  “  ““  information  clamant,  with  the  flight  control  oyatoma/auboyoteme  with  the - : - 

gossfele  specificity.  For  example:  Hydraulic  pressure  might  correlate  wife  fee  hydraulic  subsystem  and  not  fee  electrical. 

This  means  we  need  to  identify  any  necessary  relationships  between  infoimation  elements  (including  I  &  W,  measures,  etc )  and  steps  in  fee 
maintenance  procedure.  Phrased  another  way,  we  need  to  lay  out  a  map  for  what  needs  to  be  known  when. 

This  means  we  need  to  identify  how  information  elements  map  onto  fee  set  of  personnel  (either  singly  or  collectively).  Phrased  another  wav 
we  need  to  lay  out  a  map  for  who  needs  to  know  what 


174 


The  recommended  starting  point  will  be  to  offer  a  process  or  sequence  schema  to  the  SMEs  for  then- 
inspection  and  review.  This  will  be  an  illustrative  “map”  for  a  generalized  maintenance  task.  The  notional 
initial  such  “map”  is  as  follows: 

♦  Problem  Identification 

♦  Problem  Reporting/Documentation 

♦  Unit  Coordination  of  maintenance  Status  for  Aircraft 

♦  Maintenance  Setup/Preparation  (for  Handling  the  Problem) 

♦  Diagnosis 

♦  Prognosis  (Decision  on  Reparability  of  Aircraft;  Whether  to  Proceed) 

♦  Solution  Specification  (Decisions  on  What  to  Do) 

♦  Repair  Actions/Activities 

♦  Supply/Requisition  Activities 

♦  Completion  Decision  (i.e.,  deciding  when  it’s  finished) 

♦  Testing/Evaluation  of  Solution 

♦  Certification/V alidation  of  Outcomes 

♦  Solution  Reporting/Documentation 

♦  Maintenance  Stand-Down  (Cleaning  up;  clearing  out) 

♦  Unit  Coordination  of  Aircraft  Return  to  Duty 


First,  we  need  to  run  this  “map”  past  the  SMEs  to  see  if  they  believe  it’s  representative  of  a  single  pass 
through  a  maintenance  solution  path.  If  not,  it  must  be  modified  until  they’re  comfortable  with  it. 

Criteria  for  Completion: 

This  phase  is  reasonably  complete  at  the  point  we  have  obtained  a  consensus  on  the  basic  steps  or  phases 
for  a  representative  process  map 

Thing s  to  Watch  out  for  (General): 

We  need  to  pay  special  attention  to  the  following  things  (if  and  when  they  pop  up): 

♦  Specific  references  to  procedural  breakdowns  (situations  or  conditions  when  the  process  flow  is 
disrupted  or  blocked,  etc.). 

♦  Specific  references  to  breakdowns  in  team  coordination,  administrative  requirements,  etc.  (i.e., 
overhead  obstacles  to  getting  the  job  done). 

♦  Specific  differences  in  opinion  (or  mention  of  multiple  options)  in  the  general  process  schema. 
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Things  to  Watch  out  for  (for  RADSS): 

With  regard  to  the  RADSS-based  “back  end”  to  this  project,  we  need  to  pay  special  attention  to  the 
following  things  (if  and  when  they  pop  up): 

♦  Altematives/options  in  procedures 

♦  Selection  criteria  for  choosing  among  alternative  procedures 

♦  Any  mention  of  amounts,  numbers,  relative  rankings,  value  judgments,  etc.,  which  indicate 
qualitative/quantitative  metrics  relevant  to  tool  and  procedural  assessment 

HLB.  Populating  the  Process  Schema  with  Data  on  the  Four  Aspects 

Second,  we  then  need  to  “drill  down”  into  each  of  the  steps  outlined  in  the  notional  process  map.  For  each 
step,  we  need  to  identify  and  capture  data  pertaining  to  the  4  topical  aspects  outlined  earlier. 

Criteria  for  Completion: 

This  phase  is  reasonably  complete  at  the  point  we  have  obtained: 

1 .  At  least  one  pass  through  the  schema  or  map  capturing  data  on  functions 

2.  At  least  one  pass  through  this  map  capturing  data  on  tools/instruments 

3.  At  least  one  pass  through  this  map  capturing  data  on  information  requirements 

Things  to  Watch  out for  (General): 

We  need  to  pay  special  attention  to  the  following  things  (if  and  when  they  pop  up): 

♦  Specific  references  to  procedural  breakdowns  (situations  or  conditions  when  the  process  flow  is 
disrupted  or  blocked,  etc.). 

♦  Specific  references  to  informational  breakdowns  or  deficiencies  (misinterpretations;  gaps  in  Hqta; 
time  lost  learning  something,  etc.). 

♦  Specific  references  to  breakdowns  in  tool  usage  (problems,  constraints,  time  sinks,  etc.). 

♦  Specific  references  to  breakdowns  in  team  coordination,  administrative  requirements,  etc.  (i.e., 
overhead  obstacles  to  getting  the  job  done). 

Things  to  Watch  out for  (for  RADSS): 

With  regard  to  the  RADSS-based  “back  end”  to  this  project,  we  need  to  pay  special  attention  to  the 
following  things  (it  and  when  they  pop  up): 

♦  Specific  references  to  tools  and  instruments  in  use 

♦  Specific  references  to  tools  and  instruments  available,  but  not  used 
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♦  Specific  references  to  tools  and  instruments  formerly  used 

♦  Altematives/options  in  procedures 

♦  Selection  criteria  for  choosing  among  alternative  procedures 

♦  Altematives/options  in  tools/instruments  and  their  application 

♦  Selection  criteria  for  choosing  among  alternative  tools 

♦  Distinctions  between  hardware  and  software  tools 

♦  Distinctions  between  tools  used  for  (e.g.)  manipulating  materials  versus  those  used  for  measurement, 
etc. 

♦  Correlation  of  particular  tools  with  particular  phases  or  steps  in  the  process  map 

♦  Correlation  of  particular  tools  with  particular  roles  or  individuals 

♦  Any  mention  of  amounts,  numbers,  relative  rankings,  value  judgments,  etc.,  which  indicate 
qualitative/quantitative  metrics  relevant  to  tool  and  procedural  assessment 

The  above-cited  steps  are  the  basic  cores  of  the  on-site  SME  KA  effort.  They  are  the  minimum  that  we 
must  do  to  proceed  toward  RADSS  evaluation  of  tools.  I’m  not  saying  this  is  all  we  can  aspire  to  do  out  at 
Nellis.  I  am,  however,  saying  this  is  the  minimum  we  need  to  accomplish  for  each  of  the  3  maintenance 
teams/units  with  whom  we’ll  be  visiting. 

III.C.  Auxiliary  On-Site  Data  Gathering  Activities 

There  is  a  variety  of  other  data,  which  might  be  accessible  while  we’re  on-site.  If  we  can,  we  should  make 
a  point  to  gather  any  data  (no  matter  how  general)  on  the  following  topics: 

♦  Maintenance  operations  statistics  (overall  for  the  unit) 

♦  Maintenance  functions  statistics  (overall,  and  especially  for  flight  control  issues) 

♦  Incidence  statistics  for  flight  control  problems  (e.g.,  how  often,  how  long  it  takes,  etc.) 

♦  Data  on  specific  tools  (especially  those  employed  automation,  programming,  and/or  resident 
software) 

♦  Inventory  of  information  resources  available  to  the  maintenance  team 

♦  Data  on  the  maintenance  team’s  supply  chain  and  supply  procedures 

♦  Data  on  any  recent  changes/innovations  in  flight  control  maintenance  ops 

♦  Data  on  any  pending  changes/innovations  in  flight  control  maintenance  ops 


Appendix  D:  SME  Sign-in  Sheet  with  Privacy/Disclosure 
Statement 


Table  34:  Subject  Matter  Expert  Sign-in 


NAME/RANK/UNIT 


ROLE 

(Check  all  that  apply) 


♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Other( _ ) 


♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Other( _ ) 


♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Other( _ ) 


♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Other/ _  ) 


Maintenance/TECH  SPECIALIZATION 
if  applicable  -  hydraulics,  avionics,  etc. 
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♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Otherf  ) 

♦  Fault  Diagnosis 

♦  Repair 

♦  Test/Evaluation 

♦  Maintenance 
Oversight/Mgmt 

♦  Otherf  ) 

PRIVACY/DISCLOSURE  STATEMENT:  Your  participation  in  this  knowledge  acquisition  effort  has  been 
solicited  to  help  us  understand  flight  control  maintenance  functions  and  to  compile  criteria  for  evaluating  potential 
innovations  and  aids  for  these  functions.  The  solicitation  of  personal  identification  information  above  is  only  for  the 
use  of  the  TRS  D024  researchers  (for  compiling  and  organizing  our  results).  You  will  not  be  personally  identified  in 
any  reports  or  other  products  of  this  TRS  DO  24  research  effort.  The  TRS  DO  24  research  staff  will  not  disseminate 
any  personalty-identifiable  data  we  collect.  Thank  you  for  your  participation. 
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Appendix  E:  MXM  Tool/instrument  Survey  Form 


Table  35:  Tool/Instrument  Summary  Sheet 


NAME 


DESCRIPTION 

(Size,  weight,  etc.) 


TYPE  of  device 

♦  Physical  Hardware 

♦  Electrical 

Hardware 

♦  Electronic 
instrument 

♦  Programmable 
device  (PC; 
calculator,  etc.) 

♦  Software 

Application 

WHAT  it’s  used 
for: 

♦  Physical  Action 

♦  Measurement 

♦  Work  environment 
(e.g.,  stands, 
lights) 

♦  General  Reference 
(e.g.,  manual) 

♦  Decision  Aid  (e.g., 
diagnostic  guide) 

WHEN  Is  it 
needed: 

(During  what 

phase/task/ 

activity) 

♦  Fault  Discovery 

♦  Fault  Reporting 

♦  Fault  Diagnosis 
(Testing/ 
Exploration) 

♦  Fault  Diagnosis 
(Troubleshooting/ 
Decision  Aid) 

♦  Solution 

Specification 

♦  Repair 

♦  Test/Evaluation  of 
Repair  Outcomes 

♦  Maintenance 
Management 
(Documentation; 
Coordination) 

AVAILABILITY 

(Access  to  Tool) 

Readily/On-hand 

♦  Go  get  it  (within 
area) 

♦  Go  get  it 
(elsewhere) 

ACCESSIBILITY 

(Access  to  Tool) 

1  have  one  anytime 

Must  share  it  within 
unit: 

Documentation 

Communications 

Other: 


♦  Communication 
(within  team; 
across  shift,  etc.) 

♦  Supply/Requisition 


♦  Other: 

♦ 


♦  Send  out  for  it 
nearby 

♦  Send  for  it 
(Distant) 


Must  share  it  with 
other: 
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AVAILABILITY  TIME  (min/max) 

(Measures  for 
what  it  takes  to  get 
tool) 


DISTANCE  (min/max)  HOW  OFTEN? 


PORTABILITY 

(Deploy; 

maneuver) 


ENGAGEMENT 

(Deploy;  bring  to 
bear) 


Hands-Free  (can  carry  ♦  One-handed  carry 

it  on  belt  in  pocket...)  _ 

♦  Two-handed  carry 


♦  No  setup  required  ♦  Must  connect  to 

A/C 

♦  Must  set  up 

tool/instrument  ♦  Must  connect  to 

other 

unit: _ 


♦  Roll  around 
(Manual) 

♦  Roll  around 
(Driven) 


♦  Must  connect  to 
power 

♦  Must  connect  to 
other 


ENGAGEMENT 

(Physical  Use) 


ENGAGEMENT 

(Informational 

Use) 


ENGAGEMENT 

(Configuration) 


HOW  OFTEN 
EMPLOYED: 


LONGEVITY: 


EASE  OF  USE 


WHO  USES  IT: 

(flight  control 
maintenance 
team) 

PLEASE 

SPECIFY; 


♦  Hands-Free  (don’t 
have  to  manipulate 
it) 

♦  (Someone  else 
does) 

♦  One-handed 
interaction 

♦  Two-handed 
interaction 

♦  Must  read/check 
setting 

♦  Must  make 
setting(s) 

♦  Attention-Free 

♦  Must  read  data 

♦  Must  navigate 

♦  Must  select  setting 

♦  Must  enter  data 

♦  Must  translate 
data 

♦  Unchanging/“As 

Is” 

♦  MUST  reconfigure/ 
reset  during  use 

Repair  on  breakdown 

♦  Must  reconfigure/ 
reset  (once)  to  use 

♦  MAY  do  so  during 
use 

♦  0-20%  of 
incidents 

♦  40 -60%  of 
incidents 

100%  of  incidents 

♦  20-40%  of 
incidents 

♦  60  -  80%  of 
incidents 

Dispose  after  1  use 

Dispose  on  breakdown 

Repair  on  breakdown 

DIFFICULT 

NOMINAL 

EASY 

One  person  only 

Some  team  members 

All  team  members 
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ALTERNATIVES? 

(To  this  Toot) 

Yes - > 

No 

What? 

OTHER  TOOLS  THAT 
MUST  BE  USED  ALONG 
WITH  THIS  ONE? 

Yes - » 

No 

What? 

SPECIFIC  GRIPES? 

Yes - 

No 

What? 

SPECIFIC 

IMPROVEMENTS? 

Yes - 

No 

What? 

GENERAL  QUESTIONS: 

When  (in  what  circumstances)  do  you  expect  to  use  this  tool/instrument? 
When  do  you  not  expect  to  use  it  (but  you’ve  used  it  anyway)? 


Are  there  any  tools/instruments  you  expect  to  use  before  using/needing  to  use  this  one?  If  so,  what: 
Are  there  any  tools/instruments  you  expect  to  use  after  using/needing  to  use  this  one?  If  so,  what: 
What  indications  cue  you  that  you’ll  be  needing  to  employ  this  tool/instrument? 


Do  you  know  of  a  specific  tool/instrument  (single  one;  category;  type)  that’s  better  for  the  given  function 
than  this  one?  If  so,  why? 
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Appendix  F:  Flight  Control  Maintenance  Tool  Inventory:  F-15 
Shops 


This  appendix  presents  the  results  obtained  from  the  tool  survey  with  the  F-15  shops  at  Nellis  AFB.  Table 
36  lists  the  tools  and  instruments  identified  in  the  F-15E  support  section.  Table  37  lists  the  tools  and 
instruments  identified  in  the  F-15C  support  section. 


Table  36:  Strike  Shop  (F-15E)— Tool  Inventory  Data 


TTU-205  Tester 
(a.k.a.:  “TT-205,"  “205”) 

Description 

Notes/Comments 

♦  Model  Designation:  TTU-205F 

♦  Title:  “Test  Set,  Pressure  - 
Temperature" 

♦  Kit  volume:  3  cubic  feet 

♦  Kit  weight:  116  lbs. 

♦  Listed  Cost:  $35,000.00 

♦  Stock  Number:  4920-01  -214-241 0  or 
4920-00-731-1457 

♦  Part  Number:  189 104  80001 

♦  Very  commonly  employed  by  impound  team  for  flight 
control  problems. 

♦  There  are  supposed  to  be  2  available 

♦  Typically,  1  is  functional  and  1  is  in  the  shop  at  any 
given  time. 

♦  “By  the  book”  procedure  requires  205  usage  anytime 
lines  are  connected/disconnected. 

♦  A  205  is  typically  checked  out  for  usage  1  or  2  times 
per  week. 

♦  In  a  “worst  case”  scenario,  tracing  a  leak  may  require 
using  the  205  for  circa  2  workdays. 

Connection  Set  forTTU-205  Tester 
(a.k.a.:  “hose  set,"  “connector  set/kit”) 

Description 

Notes/Comments 

♦  Model  Designation:  None 

♦  Title:  “Pitot  Connection  Set” 

♦  Kit  size:  Housed  in  an  aluminum  case 
approximately  the  size  of  a  24” 
suitcase 

♦  Kit  weight:  ca.  25  lbs. 

♦  Listed  Cost:  Unknown 

♦  Connectors  and  hoses  are  problematical 

♦  It  often  seems  that  the  connection  kit  is  the  main 
constraint  in  using  the  205 
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Digital  Multimeter 

Description 

Notes/Comments 

♦  Model  Designation:  None 

♦  (Commercial  multimeter  unit) 

♦  Employed  in  troubleshooting  wiring 

♦  Used  to  check  for  continuity,  grounding,  and  resistance 
in  wiring  circuits 

Heat  Guns 

Description 

Notes/Comments 

♦  Hapco  HT900  “All  Purpose  Heat  Gun” 

♦  (Commercial  unit) 

None 

Wire  Repair  Kit 
(Commercial) 

Description 

Notes/Comments 

♦  Daniels  Mfg.  Corp.  (DMC)  is  the 
typical  supplier 

♦  Title:  “DMC  712  Maintenance/Repair 

Kit  fbrF-15” 

♦  2  such  kits  were  on  the  shelf. 

♦  Each  kit  includes  circa  40  special  dies  for  wire 
connection  work. 

Wire  Repair  Kit 
(Locally-Devised) 

Description 

Notes/Comments 

♦  Locally  developed  and  assembled. 

♦  Formally  listed  among  inventory  items. 

♦  Title:  “57th  AGS  Strike  AMF 

NJSS00822  Wire  Repair  Kit” 

♦  Total  constituent  components  =  74 

♦  These  components  are  stored  in  5  trays  plus  general 
storage  within  the  case. 
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Table  37:  F-15C  Shop  Tool  Inventory  Data 


FLTS  Tester 

Description 

Notes/Comments 

♦  Model  Designation:  AN/ASM-497 

♦  Title:  “Test  Set,  Automated  Flight 
Control  System” 

♦  Manufacturer:  GE 

♦  Listed  Cost:  $143,546.00 

♦  2  FLTS  test  units  are  in  stock. 

♦  At  the  time  of  our  visit,  one  of  these  units  had  just 
returned  from  being  serviced. 

♦  Each  FLTS  test  set  includes  a  connector  kit  (a 
sizeable  satchel  containing  cables). 

♦  There  are  (4)  cables  (W2,  W3,  power,  ground) 
which  must  be  connected  to  use  the  FLTS  tester. 

Time  Domain  Reflectometer 

(a.k.a.:  “TDR”) 

Description 

Notes/Comments 

♦  Model  Designation:  Tektronix  1502B 
TDR 

♦  Listed  Cost  Unknown 

TDR  is  employed  to  evaluating  the  distance  along  a 
conductive  cable  or  wire  to  an  apparent  break  or  gap. 

TDR  Adapter  Kit84 
(Locally  Assembled) 

Description 

. . . . 

Notes/Comments 

♦  Designation:  “TDR  Adapter  Kif 

♦  ID:  NJSE8O102 

♦  Used  to  attach  and  use  the  Tektronix  TDR. 

♦  Consists  of  a  collection  of  cables  and  adapters 
housed  in  a  circa  18”-  long  hip  roof  toolbox. 

Mach  Number  Simulator 

Description 

Notes/Comments 

♦  Manufacturer:  Pratt  & 

Whitney/Howell  Instruments 

♦  Weight:  51.9  lbs. 

♦  Size:  circa  14”  X 18"  X 18” 

♦  Listed  Cost:  $7416.00 

♦  Octal  data  output 

♦  Used  to  check  pitch  and  roll  channel  assembly  (an 
electromechanical  valve  system). 

♦  Last  checked  out  27  months  ago. 

14  The  kit  consists  of  a  total  of  24  components  (ail  lead  cables  or  adapters).  This  is  one  of  only  2  locally  devised  tools  encountered  at  Nellis. 


AFCS  Breakout  Box 

Description 

Notes/Comments 

♦  Manufacturer:  McDonnell  Douglas 

♦  Title:  “Automated  Flight  Control 
System  Breakout  Box  -  F-15” 

♦  Listed  Cost:  $7714.00 

♦  Was  last  checked  out  on  1 7  July  2002. 

♦  This  was  the  only  recorded  usage  since  1 998. 

Miscellaneous  Stands/Supports 

Description 

Notes/Comments 

♦  Stepladders 

♦  Stands 

♦  Boarding  ladder 

Miscellaneous  equipment  used  to  access  the  A/C 
during  maintenance. 

Wire  Repair  Kit 
(Locally-Devised) 

Description 

Notes/Comments 

♦  Locally  developed  and  assembled. 

♦  Formally  listed  among  inventory 
items. 

♦  Title:  “57th  AGS  Strike  AMF 
NJSS00822  Wire  Repair  Kit” 

♦  Total  constituent  components  =  74 

♦  These  components  are  stored  in  5  trays  plus 
general  storage  within  the  case. 
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Appendix  G:  Flight  Control  Maintenance  Tool  Inventory:  F-16 

This  appendix  presents  the  results  obtained  from  F-16  tool  inventory  at  Nellis  AFB. 


Table  38:  F-16  Shop:  Tool  Inventory  Data 


TTU-205  Tester 
(a.k.a.:  “TT-205,”  “205’’) 

Description 

Notes/Comments 

♦  Model  Designation:  TTU-205F 

♦  Title:  “Test  Set,  Pressure  - 
I  Temperature” 

♦  No  additional  data  or  facts  obtained  relative  to 
what  we’d  already  gathered  from  the  F-15  support 
sections. 

♦  Refer  to  the  F-1 5  tool  survey  data  for  more  details 
on  the  TT-205 

Connection  Set  forTTU-205  Tester 
(a.k.a.:  “hose  set,”  “connector  set/kit”) 

Description 

Notes/Comments 

♦  Model  Designation:  None 

♦  Title:  “Pitot  Connection  Set” 

♦  No  additional  data  or  facts  obtained  relative  to 
what  we’d  already  gathered  from  the  F-15  support 
sections. 

♦  Refer  to  the  F-15  tool  survey  data  for  more  details 
on  the  TT-205  connector  kit 

Digital  Multimeter 
(a.k.a.:  “fluke  meter”) 

Description 

Notes/Comments 

♦  Model  Designation:  None 

♦  (Commercial  multimeter  unit) 

♦  Stock  number:  6625  01  147  6182 

♦  Employed  in  troubleshooting  wiring 

♦  Used  to  check  for  continuity,  grounding,  and 
resistance  in  wiring  circuits 

♦  Three  in  stock  in  the  support  section 
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EDNA 


(Enhanced  Diagnostic  Aid) 


Description 

Notes/Comments 

♦  Manufacturer:  Lockheed  Martin/ 

Ft  Worth 

♦  Form:  Specialized  laptop-style 
hardware  with  associated  cables. 

The  EDNA  package  includes  a 
ruggedized  removable  hard  drive, 
keyboard,  and  an  internal  power 
source. 

♦  Listed  Cost:  $63,463.00 

♦  Some  cables  are  included  in  the  basic  EDNA  kit 

♦  A  second/larger  box  associated  with  the  EDNA 
contained  an  estimated  15-20  cables  of  various 
types  for  connecting  EDNA 

♦  Because  most  data  can  be  obtained  (as  Hex 
codes)  from  the  onboard  systems,  EDNA  isn’t 
used  all  the  time. 

♦  EDNA  is  typically  brought  in  for  a  “brain  bender” 

(i.e.,  a  particularly  difficult  diagnostic  problem). 

♦  EDNA  allows  the  maintainer  to  read  BIT  codes  as 
one  proceeds  with  the  troubleshooting 

Wire  Repair  Kit 
(Commercial) 

Description 

Notes/Comments 

♦  Daniels  Mfg.  Corp.  (DMC)  is  the 
supplier 

♦  Title:  “DMC  716 

Maintenance/Repair  Kit  for  F-16” 

♦  One  kit  on  the  shelf 

♦  This  DMC  kit  appeared  to  be  pretty  much  the 
same  general  set  as  the  712  kit  tailored  to  the  F- 
15s. 

♦  The  DMC  kit  is  not  often  checked  out/used. 

Time  Domain  Reflectometer 

(a.k.a :  "TDR”) 

Description 

Notes/Comments 

♦  Model  Designation:  Tektronix  1502B 
TDR 

♦  Stock  Number  6625  01  0035561 

♦  Listed  Cost:  Unknown 

♦  TDR  is  employed  to  evaluating  the  distance  along 
a  conductive  cable  or  wire  to  an  apparent  break  or 
gap. 

♦  This  unit  appeared  identical  to  the  TDRs 
employed  at  the  F-15  unit. 

♦  No  adapter  kit  had  been  locally 
developed/assembled,  as  had  occurred  over  at 
the  F-15  unit 
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Crossover  Cable  Sets 

Description 

Notes/Comments 

Designation:  None  -  just  “crossover 
cables"  or  “crossover  cable  sets” 

♦  These  cables  allow  maintained  to  temporarily 
patch  flight  control  circuits  over  to  the  other  side  of 
the  A/C. 

♦  Running  diagnostic  tests  double-checking  similar 
behavior  on  the  other  side  of  the  A/C  allows 
maintained  to  validate  apparent  fault  conditions 
and  help  figure  out  fault  locations. 

♦  One  crossover  cable  set  observed  on  the  shelf. 

Gyro  Poiarity  Cable  Set 

Description 

Notes/Comments 

♦  Designation:  None  found 

♦  Stock  number  16U14558LI-1 

♦  This  is  a  cross-connector  kit  used  to  troubleshoot 
the  gyros  on  the  F-16. 

♦  One  such  cable  set  observed  on  the  shelf. 

189 


Appendix  H:  Flight  Control  maintenance  Tool  Inventory:  C-17 

This  appendix  presents  the  results  obtained  from  the  C-17  tool  inventory  at  Charleston  AFB. 


Table  39:  C-17  Tool  Inventory  Data 


TTU-205  Tester 
(a.k.a.:  “TT-205,"  “205”) 

Description 

Notes/Comments 

♦  Model  Designation:  TTU-205F 

♦  Title:  “Test  Set,  Pressure  - 
Temperature” 

♦  No  additional  data  or  facts  obtained  relative  to  what  we’d 
already  gathered  from  the  F-15  support  sections. 

♦  Refer  to  the  F-1 5  tool  survey  data  for  more  details  on  the 
TT-205 

♦  A  total  of  three  in  stock 

♦  Model  Designation:  TTU-205D 

♦  Title:  “Test  Set,  Pressure  - 
Temperature" 

♦  No  additional  data  or  facts  obtained  relative  to  what  we’d 
already  gathered  from  the  F-15  support  sections. 

♦  Refer  to  the  F-1 5  tool  survey  data  for  more  details  on  the 
TT-205 

♦  A  total  of  two  in  stock 

Connection  Set  forTTU-205  Tester85 
(a.k.a.:  "hose  set,”  “connector  set/kit") 

Description 

Notes/Comments 

♦  Model  Designation:  None 

♦  Title:  “Pitot  Connection  Set” 

♦  Refer  to  the  F-1 5  tool  survey  data  for  more  details  on  the 
TT-205  connector  kit 

♦  Six  in  stock 

^  V  305  00~'0“to’-  «"*  A.  ototod  t  M.llic,  th.  o.to  moro  ofU„  probl.m.ti.cJ 

Aan  the  testers  themselves.  The  hose  sets  have  to  be  ordered  from  Canada.  In  use,  there  are  2  hoses  that  have  to  be  attached  to  the  TT-205  tester. 
The  most  common  failure  point  is  not  the  hose,  but  the  seals  in  the  connectors.  These  seals  have  a  tendency  to  warp  or  “roll  up”  within  the 

P  1 bod'f'  Anothfr  Pr?blem  to  the  C-l  7  is  hose  length.  It  was  noted  that  new  hoses  recently  requisitioned  could  not  be  used  on  the 

C-l  7  because  they  were  too  short  to  permit  the  required  connections  between  the  aircraft  and  the  tester  unit. 
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Inclinometers88 


Description 

Notes/Comments 

Model  Designation:  Hilger  &  Watts 
T8108-1 

♦  Employed  in  checking  flight  surfaces’  and  other 
components’  alignment/orientation 

♦  Analog  type 

♦  3  in  stock 

Model  Designation:  Hilger  &  Watts 

Model  ATB107 

♦  Employed  in  checking  flight  surfaces’  and  other 
components’  alignment/orientation 

♦  Analog  type  -  larger  than  the  T81 08-1 

♦  1  in  stock 

Digital  Multimeter 
(a.k.a.:  “fluke  meter”) 

Description 

Notes/Comments 

♦  Model  Designation:  Fluke  77111 

♦  (Commercial  multimeter  unit) 

♦  Employed  in  troubleshooting  wiring 

♦  Used  to  check  for  continuity,  grounding,  and  resistance  in 
wiring  circuits 

♦  One  found  in  the  support  section 

♦  Model  Designation:  Fluke  8025B 

♦  (Commercial  multimeter  unit) 

♦  Usage  same  as  noted  above  for  771 II 

♦  One  found  in  the  support  section 

DTOS 

Description 

Notes/Comments 

♦  Computer-based  reference  and 
diagnostic  aid  for  C-17 

♦  Platform:  Panasonic  “Toughbook” 
(ruggedized  laptop),  which  costs 
circa  $6500.00. 

♦  Listed  Cost:  $63,463.00 

♦  CD-based  software  package  providing  wide-ranging 
technical  documentation  on  C-17  maintenance  issues.87 

♦  Main  features  noted  were  Job  Guide  documentation  and 
wiring  diagrams. 

♦  For  both  text  and  diagrams,  on-screen  legibility  required 
zoom  factor  of  at  least  125%. 

♦  At  1 00%  and  125%  zoom  factors,  neither  text  nor 
diagram  displays  could  be  shown  within  the  bounds  of 
the  available  display  screen. 

86  Only  “analog”  inclinometers  were  on  hand,  but  newer  digital  inclinometers  were  on  order. 

87  The  variety  of  reference  materials  available  is  indicated  in  the  main  index  entries,  which  include:  Fault  Isolation;  Flight  Manual  Supplements; 
FMs;  General  Reference  Manuals;  General  Systems;  Inspection  Manuals  and  Workcards;  Interface  Test  Adapters;  Intermediate  Maintenance 
Instructions;  IPBs;  Job  Guides,  Maintenance  Manual  Supplements;  Schematic  Diagrams;  Structures;  Support  Equipment;  TCTOs;  and  Wiring 
Diagrams. 
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Wire  Repair  Kits/Equipment88 


Description 

Notes/Comments 

♦  Daniels  Mfg.  Corp.  (DMC)  is  the 
supplier 

♦  Title:  “DMC  31  C-141/CHS8/H003 

One  kit  on  the  shelf 

♦  “Pin  Kit -Install" 

♦  Supplier:  Astro  Tool  Co. 

♦  Part  Number:  M83521/6-01 

One  kit  on  the  shelf 

♦  “Flap  Pigtail” 

♦  Stock  Number  1680  P02147  24418 

♦  Part  Number:  1 7P6E  41 50-502 

♦  One  on  the  shelf 

♦  Used  in  troubleshooting  flaps 

Time  Domain  Reflectometer 

(a.k.a.:  “TDR") 

Description 

Notes/Comments 

Model  Designation:  Tektronix  1502B 

TDR 

♦  TDR  is  employed  to  evaluating  the  distance  along  a 
conductive  cable  or  wire  to  an  apparent  break  or  gap. 

♦  Two  units  in  stock  at  the  support  section 

♦  No  adapter  kit  had  been  locally  developed/assembled,  as 
had  occurred  at  the  Nellis  AFB  F-15  unit. 

Cable  Breakout  Boxes 

Description 

Notes/Comments 

♦  Designation:  “Cable  Breakout  Box” 

♦  Part  Number  1 7G41 0523  1 

Two  in  stock 

Rigging  Equipment/Implements 

Description 

Notes/Comments 

♦  “Protractor  Set  -  Rigging” 

♦  Part  Number:  1 76-1 40307-23 

One  in  stock 

♦  “Protractor  Set  -  Rigging” 

*  Pent  Number.  170-14100*4-1 

One  in  stock 

"  The  support  section  staff  stated  that  most  individual  wire  repair  implements  (e.g.,  crimpers,  etc.)  were  separately  binned  and  inventoried. 
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♦  “Rigging  Equipment  -  Cable 
Mechanical” 

♦  Supplier  McDonnell  Douglas 

♦  Part  Number:  176-140015-1A 

♦  Rigging  implements  for  fixing  mechanical  controls  on  C- 
17 

♦  Two  in  stock 

♦  “Rigging  Equipment  -  Cable 
Mechanical” 

♦  Supplier.  McDonnell  Douglas 

♦  Part  Number  176-140015-1 

♦  Rigging  implements  for  fixing  mechanical  controls  on  C- 
17 

♦  One  in  stock 

♦  Part  number  differs  from  previous  item  by  one  letter  (“A" 
at  the  end).  Unable  to  find  out  what  differentiates  the  2 
sets. 

Miscellaneous  Rigs  and  Fixtures 

Description 

Notes/Comments 

Stands  and  structural  rigs 

Miscellaneous  equipment  used  to  access  the  A/C  during 
maintenance  and/or  support  items 
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Appendix  I:  Logistics  Quality  Performance  Measures: 
Fighters  -  FY1993  through  FY2002 


One  of  the  KA  goals  was  to  obtain  general  data  on  maintenance  operations.  In  the  course  of  the  KA  visits 
a  point  was  made  to  solicit  statistical  data  on  maintenance  performance  for  those  aircraft  included  in  the 
KA.  Among  the  data  collected  were  a  series  of  ten-year  summaries  for  maintenance  on  the  operational 
fighters  covered  in  the  KA  efforts  and  a  month-by-month  summary  of  8-hour  fix  rates  for  the  period  April 
2002  through  March  2003.  This  data  was  obtained  and  provided  to  the  contractor  team  by  AFRL/HESR. 

The  ten-year  summary  data  has  been  transcribed  into  illustrative  tables  (Tables  40  -  46)  for  five 
categories  of  aircraft  (all  operational  fighters;  A/OA-10A;  F-15C/D;  F-15E;  and  F-16C/D).  Each  table 
offers  a  year-by-year  inventory  of  relevant  4-hour  and  8-hour  performance  standards  and  actual  recorded 
fix  rates.  For  each  of  these  metrics  the  composite  ten-year  average  is  computed.  The  deviations  between 
average  standard  and  average  fix  rate  metrics  are  computed  for  both  the  4-hour  and  8-hour  metrics. 
Finally,  the  overall  trend  (first  year  to  last  year)  is  computed  for  illustration.  Finally,  the  monthly 
summary  for  CY02  -  CY03  is  compiled  into  another  table. 
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Table  40:  Ten-year  Summary:  Fix  Rates  for  A/OA-10A  Fighters  (Total) 


Fiscal  Year 

4-Hr  Fix  Rate 
(Standard) 

4-Hr  Fix  Rate 
(Actual) 

8-Hr  Fix  Rate 
(Standard) 

8-Hr  Fix  Rate 
(Actual) 

1993 

66 

70.8 

85 

85.6 

1994 

66 

70.1 

85 

87.2 

1995 

65 

74.4 

80 

88.6 

1996 

65 

73.4 

80 

86.1 

1997 

65 

69.8 

80 

84.9 

1998 

65 

64.7 

80 

80.0 

1999 

65 

61.9 

80 

78.8 

2000 

65 

63.1 

80 

77.8 

2001 

65 

61.8 

80 

76.6 

2002 

65 

65.2 

80 

80.1 

10-Yr  Average 

65.2 

67.5 

81 

82.57 

Deviation  from 

Standard 
(Decade  Avg.) 

+3.5% 

(better  than 
standard) 

+1.9% 

(better  than 
standard) 

10-Year  Trend 
(First  vs.  Last) 

Nominally 

Unchanged 

-7.9% 

-5.88% 

-6.4% 
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Table  41:  Ten-year  Summary:  Fix  Rates  for  F-15G/D  Fighters  (Total) 


Fiscal  Year 

4-Hr  Fix  Rate 
(Standard) 

4-Hr  Fix  Rate 
(Actual) 

8-Hr  Fix  Rate 
(Standard) 

8-Hr  Fix  Rate 
(Actual) 

1993 

57 

61.8 

75/72 

80.2 

1994 

57 

63.4 

75/72 

81.4 

1995 

57 

62 

75 

80.6 

1996 

57 

60 

75 

77.8 

1997 

57 

52.9 

75 

72.9 

1998 

57 

53.7 

75 

71.3 

1999 

57 

48.9 

75 

67.9 

2000 

57 

49.3 

75 

67.8 

2001 

57 

44.6 

75 

65.3 

2002 

57 

42.6 

75 

63.8 

10-Yr  Average 

57 

53.92 

75  (approx.) 

72.9 

Deviation  from 

Standard 
(Decade  Avg.) 

-5.4% 

(short  of 
standard) 

-2.8% 

(short  of 
standard) 

10-Year  Trend 
(First  vs.  Last) 

Unchanged 

-31.1% 

Unchanged 

-20.4% 
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Table  42:  Ten-year  Summary:  Fix  Rates  for  F-15E  Fighters  (Total) 


Fiscal  Year 

4-Hr  Fix  Rate 
(Standard) 

4-Hr  Fix  Rate 
(Actual) 

8-Hr  Fix  Rate 
(Standard) 

8-Hr  Fix  Rate 
(Actual) 

1993 

57 

63.6 

75/72 

82.1 

1994 

57 

58.7 

75/72 

78.5 

1995 

60 

60.5 

75 

79.2 

1996 

60 

57.9 

75 

77.2 

1997 

60 

50.0 

75 

70.8 

1998 

60 

48.8 

75 

68.0 

1999 

60 

50.1 

75 

69.7 

2000 

60 

51.1 

75 

71.4 

2001 

60 

48.2 

75 

68.3 

2002 

60 

47.2 

75 

66.9 

10-Yr  Average 

59.4 

53.6 

75  (approx.) 

73.2 

Deviation  from 

Standard 
(Decade  Avg.) 

-9.75% 

(short  of 
standard) 

-2.24% 

(short  of 
standard) 

10-Year  Trend 
(First  vs.  Last) 

+5.3% 

-25.8% 

Nominally 

Unchanged 

-18.5% 
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Table  43:  Ten-year  Summary:  Fix  Rates  for  F-16C/D  Fighters  (Total) 


Fiscal  Year 

4-Hr  Fix  Rate 
(Standard) 

4-Hr  Fix  Rate 
(Actual) 

1993 

66 

71.7 

1994 

66 

71.5 

1995 

66 

67.9 

1996 

66 

64.8 

1997 

66 

61.8 

1998 

66 

59.6 

1999 

66 

60.6 

2000 

66 

58.2 

2001 

66 

61.1 

2002 

66 

57.3 

10-Yr  Average 

66 

63.45 

Deviation  from 

Standard 
(Decade  Avg.) 

-3.86% 

(short  of 
standard) 

10-Year  Trend 
(First  vs.  Last) 

Unchanged 

-20.1% 

8-Hr  Fix  Rate 
(Standard) 

8-Hr  Fix  Rate 
(Actual) 

85 

89.2 

85 

88.6 

85 

84.7 

85 

81.2 

85 

80.2 

85 

78.2 

85 

78.1 

85 

77.3 

85 

78.7 

85 

76.6 

85 

81.28 

-4.38% 

(short  of 
standard) 

Unchanged 

-14.13% 
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Table  44:  Ten-year  Summary:  Fix  Rates  for  All  Operational  Fighters 


Fiscal  Year 

4  Hr  Fix  Rate  (Actual) 

8  Hr  Fix  Rate  (Actual) 

1993 

63.9 

82.8 

1994 

63.7 

82.9 

1995 

63.0 

80.9 

1996 

61.9 

79.3 

1997 

57.6 

76.7 

1998 

55.7 

74.1 

1999 

54.6 

73.0 

2000 

54.0 

72.5 

2001 

52.7 

71.2 

2002 

51.4 

70.9 

10-Yr  Average 

57.85 

76.43 

10-Year  Trend  (First  vs.  Last) 

-19.6% 

-14.37% 
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Table  45:  Eight-hour  Fix  Rates  for  A/OA-IOs  &  F-15C/D  &  Es:  CY02  -  CY0389 


Month 

(CY2002  -  CY2003) 

A/Oa-10  Fleet 
Standard  =  80 

F-16C/D  Fleet 
Standard  =  76 

F»15e  Fleet 
Standard  *  75 

April 

85.3 

66.8 

64.2 

May 

80.6 

68.2 

66.7 

June 

80.7 

65.1 

69.7 

July 

78.9 

54.5 

66.5 

August 

79.8 

54.0 

60.9 

September 

80.1 

58.4 

65.7 

October 

78.1 

61.4 

66.1 

November 

81.0 

63.0 

63.4 

December 

80.7 

62.1 

62.6 

January 

81.2 

58.6 

68.8 

February 

85.9 

59.5 

75.0 

March 

85.9 

63.1 

72.4 

Cumulative 

81.6 

61.2 

67.1 

Cumulative  versus 
Standard 

+1.6% 

(above  standard) 

-18.4% 

(short  of  standard) 

-10.5% 

(short  of  standard) 

89  SOURCE:  Headquarters  ACC  Briefing:  Logistics  Maintenance  Performance  Indicators  -  Fighters  -  March  03  (Unclassified) 
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Table  46:  Eight-hour  Fix  Rates  for  F-16C/D,  Blocks  30, 40  &  50:  CY02  -  CY0390 


MONTH 

F-16C/D  FLEET 

F-16C/D  FLEET 

F-16C/D  FLEET 

(CY2002  -  CY2003) 

(Block  30) 

(Block  40) 

(Block  50) 

Standard  «*  85 

Standard  =  85 

Standard  »  85 

April 

76.6 

67.9 

87.2 

May 

72.2 

68.2 

87.2 

June 

66.1 

70.6 

81.6 

July 

67.9 

62.3 

80.4 

August 

60.2 

68.2 

86.3 

September 

75.0 

72.4 

82.4 

October 

74.0 

71.3 

85.6 

November 

82.1 

70.7 

87.9 

December 

84.5 

68.1 

77.9 

January 

76.1 

71.2 

77.9 

February 

86.9 

67.5 

72.1 

March 

80.9 

71.6 

87.8 

Cumulative: 

74.9 

69.3 

83.2 

Cumulative  versus 

-11.9% 

-18.5% 

-2.1% 

Standard 

(short  of  standard) 

(short  of  standard) 

(short  of  standard) 

90  SOURCE:  Headquarters  ACC  Briefing:  Logistics  Maintenance  Performance  Indicators  -  Fighters  -  March  03  (Unclassified) 
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Appendix  J:  Guidelines  for  Subsystem  Selection 

Tables  47  -  54  depict  the  specific  data  supporting  the  subsystem  selection  decision  process. 


Table  47:  Fuel 


FUEL 

A-10 

B-1 

B-52 

F-16 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

5 

5 

5 

5 

5 

5 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

2 

5 

5 

3 

3 

4 

Relevance  to  MX  Mentor 
project 

(1  -Not  much  5-Significant) 

1 

1 

1 

1 

2 

1 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

1 

2 

1 

1 

2 

1 

Potential  Improvement  for 

Maintainer 

(1-Little  5-Major) 

2 

5 

3 

3 

3 

3 

TOTAL-86 

11 

18 

15 

13 

15 

14 
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Table  48:  Hydraulic 


HYDRAULIC 

A-10 

B-1 

B-52 

F-15 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

1 

1 

1 

1 

1 

1 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

2 

4 

3 

3 

1 

3 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

1 

1 

1 

1 

1 

1 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

2 

1 

1 

2 

3 

1 

Potential  Improvement  for 

Maintainer 

(1-Little  5-Major) 

1 

3 

3 

2 

1 

3 

TOTAL-51 

7 

10 

9 

9 

7 

9 

Table  49:  Propulsion 


PROPULSION 

A-10 

B-1 

B-S2 

F-15 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

5 

5 

1 

5 

5 

5 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

1 

5 

3 

4 

4 

2 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

3 

5 

1 

4 

4 

3 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

3 

1 

1 

2 

2 

3 

Potential  Improvement  for 
Maintainer 

(1.1  ittla  5-Major) 

2 

4 

1 

3 

4 

3 

TOTAL-94 

14 

20 

7 

18 

19 

16 
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Table  50:  Landing  Gear 


LANDING  GEAR 

A-10 

B-1 

B-52 

F-15 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

5 

1 

5 

5 

1 

5 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

1 

5 

5 

3 

2 

3 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

1 

2 

3 

3 

1 

3 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

5 

1 

1 

3 

4 

1 

Potential  Improvement  for 

Maintainer 

(1-Little  5-Major) 

3 

2 

3 

3 

1 

3 

TOTAL-84 

15 

11 

17 

17 

9 

15 

Table  51:  Flight  Controls 


FLIGHT  CONTROLS 

A-10 

B-1 

B-52 

F-16 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

5 

5 

1 

5 

5 

1 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

1 

5 

2 

5 

3 

4 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

2 

4 

1 

5 

5 

3 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

5 

2 

2 

4 

5 

3 

Potential  Improvement  for 
Maintainer 

(1 -Little  5-Major) 

2 

4 

1 

5 

5 

3 

TOTAL— 103 

15 

20 

7 

24 

23 

14 
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Table  52:  Radar 


RADAR 

A-10 

B-1 

B-52 

F-15 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

1 

1 

1 

1 

1 

1 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

1 

3 

1 

3 

2 

1 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

1 

3 

1 

2 

1 

1 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

5 

1 

3 

3 

4 

5 

Potential  Improvement  for 

Maintainer 

(1 -Little  5-Major) 

1 

2 

1 

3 

3 

2 

TOTAL-59 

9 

10 

7 

12 

11 

10 

Table  53:  Electronic  Countermeasures 


ELECTRONIC 

COUNTERMEASURES 

(ECM) 

A-10 

B-1 

B-52 

F-15 

F-16 

EC/HC- 

130 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

1 

1 

5 

1 

5 

1 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

2 

3 

4 

2 

3 

2 

Relevance  to  MX  Mentor 
project 

(1-Not  much  5-Significant) 

1 

2 

4 

1 

4 

1 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

5 

2 

3 

3 

3 

Potential  Improvement  for 

Maintainor 

(1-Little  5-Major) 

1 

2 

3 

1 

3 

1 

TOTAL-71 

10 

9 

18 

00 

18 

8 

205 


Table  54:  Electrical 


ELECTRICAL 

A-10 

B-1 

B-52 

F-15 

Top  5  Man-hour  consumer 
(1-No  5-Yes) 

1 

5 

5 

1 

Ease  of  troubleshooting 
(1-Easy  5-Tough) 

2 

5 

5 

3 

Relevance  to  MX  Mentor 
project 

(1-Notmuch  5-Significant) 

1 

3 

3 

1 

Potential  Integration 

Challenges 

(1-Huge  5-Manageable) 

5 

1 

1 

2 

Potential  Improvement  for 

Maintainer 

(1-Little  5-Major) 

1 

5 

5 

1 

TOTAL-82 

10 

19 

19 

8 

206 


